Strings, encoding, and file I/O

Why this matters

You will read, parse, and write text every day — config, logs, CSVs, JSON, user input. Python's string handling is excellent, but only if you know the modern idioms: f-strings, pathlib, encoding-aware open, json, and the io module. This lesson installs those.

Learning objectives

Use f-strings for all formatting.
Encode and decode text correctly (UTF-8, BOM, errors policies).
Use pathlib for all file paths.
Read and write JSON, CSV, JSONL, TOML.
Work with io objects and context managers around files.

1. Strings: methods you'll actually use

python

s = "  Hello, World!  "
s.strip()                       # "Hello, World!"
s.strip(" !")                   # "Hello, World"
s.lower(); s.upper(); s.title(); s.casefold()    # casefold handles non-ASCII
s.replace("World", "Ada")
s.split(",")                    # ['  Hello', ' World!  ']
",".join(["a", "b", "c"])
s.startswith("  H"); s.endswith("!  ")
"World" in s
s.find("World")                 # index, or -1
s.index("World")                # index, raises ValueError if missing
s.count("l")

# Padding / alignment
"42".rjust(5, "0")              # "00042"
"hi".ljust(5)                   # "hi   "
"hi".center(5, "-")             # "-hi--"
"42".zfill(5)                   # "00042"

# Predicates
"42".isdigit(); "hi".isalpha(); "hi 42".isalnum()
"  ".isspace(); "Hello".istitle(); "ABC".isupper()

casefold is the right tool for case-insensitive comparisons across languages (German "ß" ↔ "ss"). Use it where you'd previously use lower() for comparison.

2. f-strings (your default since 3.6)

python

name, age, pi = "Ada", 30, 3.14159
f"Hello, {name}, age {age}"
f"pi is {pi:.2f}"               # "pi is 3.14"
f"big: {1_000_000:,}"           # "big: 1,000,000"
f"hex: {255:x}"                 # "hex: ff"
f"binary: {10:b}"               # "binary: 1010"
f"pct: {0.85:.1%}"              # "pct: 85.0%"
f"sci: {0.0001234:.3e}"         # "sci: 1.234e-04"
f"pad: {42:>5}"                 # "pad:    42"
f"date: {date.today():%Y-%m-%d}"

# Self-documenting (3.8+)
f"{name=}, {age=}"              # "name='Ada', age=30"

# Multi-line, nested quotes, expressions, comments (3.12+)
f"""
result = {sum(
    x for x in range(10)  # comment ok in 3.12
)}
"""

Format spec mini-language

{value:[fill][align][sign][#][0][width][,][.precision][type]}

Common types: s str, d int, f float, e scientific, % percent, x hex, b binary, o octal, , thousands separator.

Reach for f-strings 99% of the time. Use %-style or .format() only when the string is data (a template loaded from somewhere).

For multi-line interpolation with safer escaping, template strings (string.Template) or Jinja2 (Phase 8.1) are better choices.

3. Encoding — text vs bytes

Unicode strings (str) vs raw bytes (bytes) are different types:

python

"hello".encode("utf-8")          # b'hello'
b"\xc3\xa9".decode("utf-8")      # 'é'

len("é")                          # 1  (chars)
len("é".encode("utf-8"))          # 2  (bytes)

UTF-8 is the default everywhere in 2026 — files, network, terminals. Always specify it explicitly. The default on Windows used to be cp1252; Python 3.15 (PEP 686) makes UTF-8 mode the default everywhere.

Until then:

python

with open("file.txt", encoding="utf-8") as f:    # ALWAYS specify
    ...

Errors policies

python

b"\xff".decode("utf-8")                 # UnicodeDecodeError
b"\xff".decode("utf-8", errors="ignore")        # ''
b"\xff".decode("utf-8", errors="replace")       # '\ufffd'
b"\xff".decode("utf-8", errors="backslashreplace")  # '\\xff'

Use "ignore" only if you genuinely don't care. "replace" is good for logs and lossy display.

BOM (Byte Order Mark)

Some Windows tools save UTF-8 with a 3-byte BOM at the start. Use encoding="utf-8-sig" to handle it transparently.

4. `pathlib` — use it for all paths

pathlib.Path replaces os.path, os.walk, glob.glob, shutil.move for path manipulation. Object-oriented, OS-aware, expressive.

python

from pathlib import Path

p = Path("data/raw") / "users.csv"      # / overloaded
p.exists(); p.is_file(); p.is_dir()
p.parent                                # Path('data/raw')
p.name                                  # 'users.csv'
p.stem                                  # 'users'
p.suffix; p.suffixes                    # '.csv'; ['.csv']
p.parts                                 # ('data', 'raw', 'users.csv')
p.absolute(); p.resolve()               # absolute, symlinks resolved
p.with_suffix(".parquet")               # data/raw/users.parquet
p.with_name("other.csv")
p.relative_to(Path.cwd())

# Read / write — one-liners
text = p.read_text(encoding="utf-8")
data = p.read_bytes()
p.write_text("hello", encoding="utf-8")
p.write_bytes(b"\x00")

# Create / delete
Path("out").mkdir(parents=True, exist_ok=True)
p.unlink(missing_ok=True)               # delete file
Path("emptydir").rmdir()
import shutil
shutil.rmtree("out")                    # recursive delete

# Iteration / globbing
for f in Path("data").iterdir(): ...
for f in Path("data").glob("*.csv"): ...
for f in Path("data").rglob("**/*.json"): ...

# Useful constants
Path.cwd(); Path.home()

os.path.join("a", "b") → Path("a") / "b". Always.

5. Reading and writing files — the right way

python

from pathlib import Path

# Read all
content = Path("file.txt").read_text(encoding="utf-8")

# Read line by line (stream large files)
with open("big.log", encoding="utf-8") as f:
    for line in f:
        process(line.rstrip("\n"))

# Read with newlines preserved
with open("crlf.txt", newline="") as f:
    raw = f.read()

# Write
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("hello\n")
    f.writelines(["a\n", "b\n"])

# Append
with open("log.txt", "a", encoding="utf-8") as f:
    f.write("event\n")

File modes

Mode	Meaning
`r`	read (default)
`w`	write — truncates!
`a`	append
`x`	exclusive create — fails if exists
`b`	binary (add to any) — `"rb"`, `"wb"`
`+`	read + write

Always use with (the context manager closes the file even on exceptions).

6. JSON

python

import json
data = json.loads('{"a": 1, "b": [2, 3]}')           # str → dict
text = json.dumps(data, indent=2, ensure_ascii=False)   # dict → str

from pathlib import Path
Path("data.json").write_text(json.dumps(data, indent=2), encoding="utf-8")
data = json.loads(Path("data.json").read_text(encoding="utf-8"))

For large files, stream with json.load(file) / json.dump(obj, file).

For very large or untyped data, use msgspec (10-100× faster than json + schema validation) or orjson (fast pure-JSON).

JSONL (one JSON per line)

The default format for streaming events, logs, datasets:

python

import json
with open("events.jsonl", encoding="utf-8") as f:
    for line in f:
        event = json.loads(line)
        process(event)

# Write
with open("events.jsonl", "a", encoding="utf-8") as f:
    for ev in events:
        f.write(json.dumps(ev) + "\n")

7. CSV

python

import csv
from pathlib import Path

# Read with header
with open("users.csv", encoding="utf-8", newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["name"], row["age"])

# Write
rows = [{"name": "Ada", "age": 30}, {"name": "Bob", "age": 25}]
with open("out.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "age"])
    writer.writeheader()
    writer.writerows(rows)

newline="" is required on Windows to avoid double newlines.

For real-world CSV with messy types, switch to pandas (Phase 5.2) or polars (Phase 5.3).

8. TOML and YAML

TOML (the modern config standard, used by `pyproject.toml`)

python

import tomllib                       # 3.11+, read-only
with open("pyproject.toml", "rb") as f:
    cfg = tomllib.load(f)

To write TOML, use tomli-w or tomlkit. For full read+write, tomlkit preserves comments and formatting.

YAML

Not in stdlib. pip install pyyaml or ruamel.yaml.

python

import yaml
data = yaml.safe_load(Path("config.yaml").read_text())
Path("config.yaml").write_text(yaml.safe_dump(data, sort_keys=False))

Always safe_load, never yaml.load (the latter can execute arbitrary code).

9. `io` module — streams as objects

python

import io

# In-memory text/bytes "files"
buf = io.StringIO()
buf.write("hello")
buf.seek(0)
buf.read()                          # "hello"

bbuf = io.BytesIO(b"\x00\x01\x02")
bbuf.read(1)                        # b'\x00'

# Useful for testing functions that take a file-like object
def export(data, fh):
    fh.write("\n".join(data))

buf = io.StringIO()
export(["a", "b"], buf)
assert buf.getvalue() == "a\nb"

io is the layer behind open(). Useful for unit tests, in-memory pipelines, and when integrating with libraries that need file-like objects (e.g., csv writers).

10. stdin/stdout, argv

python

import sys

for line in sys.stdin:               # streams; one line at a time
    print(line.upper(), end="")

sys.argv                             # list of cli args; sys.argv[0] is script name
sys.exit(1)                          # exit with code
print("error", file=sys.stderr)

For CLIs use typer or click (Phase 8 / projects), not raw argv.

11. Pretty printing

python

from pprint import pp
pp(data, depth=2, width=80)

# Rich (third-party): prettier
from rich import print
print({"x": 1, "y": [2, 3]})

For interactive debugging, rich.console.Console().print(...) is unmatched.

Hands-on lab (2 hours)

Read a JSONL log file; print events filtered by a level.
Convert that JSONL into CSV; write with DictWriter.
Use pathlib.Path.rglob to find all *.py files under a directory and report total lines of code.
Read your own pyproject.toml with tomllib; print the project version.
Round-trip a UTF-8 file with BOM (utf-8-sig); preserve content.
Write a function that takes a file-like object and writes a CSV; test it with io.StringIO (no real file needed).
Bonus: chunked file copy with progress bar (rich.progress).

Common pitfalls

Forgetting encoding="utf-8" — works on macOS, breaks on Windows.
Opening a file without with — leaks the handle until GC.
Reading a huge file with f.read() instead of iterating.
Confusing str and bytes; using .decode() on a str or .encode() on bytes.
Writing CSV without newline="" → double newlines on Windows.
Using os.path in new code — switch to pathlib.
yaml.load instead of yaml.safe_load.

Self-check

Why specify encoding="utf-8" even on Linux/macOS?
Difference between str and bytes.
Why prefer pathlib over os.path?
What is JSONL?
State two reasons to use io.StringIO.

References

PEP 686 — Make UTF-8 mode default.
Fluent Python, Ramalho — Chapter 4 (text vs bytes).
pathlib docs: https://docs.python.org/3/library/pathlib.html.
json, csv, tomllib docs.
rich: https://rich.readthedocs.io/.

Why this matters

Learning objectives

1. Strings: methods you'll actually use

2. f-strings (your default since 3.6)

Format spec mini-language

3. Encoding — text vs bytes

Errors policies

BOM (Byte Order Mark)

4. pathlib — use it for all paths

5. Reading and writing files — the right way

File modes

6. JSON

JSONL (one JSON per line)

7. CSV

8. TOML and YAML

TOML (the modern config standard, used by pyproject.toml)

YAML

9. io module — streams as objects

10. stdin/stdout, argv

11. Pretty printing

Hands-on lab (2 hours)

Common pitfalls

Self-check

References

4. `pathlib` — use it for all paths

TOML (the modern config standard, used by `pyproject.toml`)

9. `io` module — streams as objects