Generators, iterators, and yield from

Why this matters

Generators are how Python handles "lazy" data: process gigabytes of logs in constant memory; build infinite sequences; turn complex producer/consumer code into simple yield statements. They underpin file iteration, comprehensions, asyncio, and pipelines. This is one of the highest-leverage Python features.

Learning objectives

Implement the iterator protocol (__iter__ / __next__).
Write generator functions and generator expressions.
Use yield from for composition.
Build streaming data pipelines.
Understand generator state and send / throw / close.

1. The iterator protocol

Two methods:

__iter__(self) -> Iterator: return an iterator over self.
__next__(self) -> T: return the next item; raise StopIteration when done.

python

class Range:
    def __init__(self, start, stop, step=1):
        self.start, self.stop, self.step = start, stop, step
    def __iter__(self):
        return RangeIter(self.start, self.stop, self.step)

class RangeIter:
    def __init__(self, start, stop, step):
        self.cur, self.stop, self.step = start, stop, step
    def __iter__(self):
        return self
    def __next__(self):
        if self.cur >= self.stop:
            raise StopIteration
        x = self.cur
        self.cur += self.step
        return x

for i in Range(0, 5):
    print(i)

Verbose. In practice, always use a generator function instead.

Iterable vs Iterator

Iterable: has __iter__; can be looped over multiple times.
Iterator: has __next__; tracks position; exhausted after one pass.

python

xs = [1, 2, 3]                # iterable
it = iter(xs)                 # iterator
next(it), next(it), next(it)  # 1, 2, 3
next(it)                      # StopIteration

A for loop calls iter(x) once, then next(...) repeatedly until StopIteration.

2. Generator functions — `yield`

A function with yield becomes a generator. Calling it returns a generator object; iterating it runs the body until each yield.

python

def countdown(n):
    while n > 0:
        yield n
        n -= 1

list(countdown(3))            # [3, 2, 1]

for x in countdown(5): print(x)

What happens:

countdown(5) returns a generator (doesn't run the body yet).
for calls next(gen). Function runs until yield n (yields 5).
Next next resumes after the yield; loop iterates; runs to next yield.
When function returns, StopIteration is raised; for exits.

State (n) persists between calls. Magical.

Reading from a file lazily

python

def lines_with(path, substring):
    with open(path, encoding="utf-8") as f:
        for line in f:
            if substring in line:
                yield line.rstrip("\n")

for ln in lines_with("big.log", "ERROR"):
    print(ln)

Processes a 10 GB file with O(1) memory.

3. Generator expressions

Same syntax as list comprehensions but with ():

python

squares = (x * x for x in range(10**9))           # lazy
sum(x * x for x in range(10**9))                  # streams; constant memory

When passed as the only argument to a function, the outer parens can be omitted:

python

sum(x * x for x in range(10))                     # OK
list(filter(lambda x: x > 0, (x*x for x in xs)))  # OK

Use generator expressions when:

The next step in the pipeline only iterates once.
The input is large or infinite.

Use a list comprehension when:

You need indexing, length, or to iterate multiple times.

4. `yield from` — composition

Delegate iteration to a sub-generator:

python

def chain(*iters):
    for it in iters:
        yield from it

list(chain([1, 2], [3, 4], [5]))       # [1, 2, 3, 4, 5]

Equivalent to for x in it: yield x for each, but also forwards .send() / .throw() / .close() correctly.

Flattening

python

def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

list(flatten([1, [2, [3, [4]]], 5]))    # [1, 2, 3, 4, 5]

5. Pipelines

Generators chain naturally:

python

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        yield from f

def parse(lines):
    for line in lines:
        yield json.loads(line)

def filter_errors(events):
    for e in events:
        if e.get("level") == "ERROR":
            yield e

def to_summary(events):
    for e in events:
        yield {"ts": e["ts"], "msg": e["msg"]}

pipeline = to_summary(filter_errors(parse(read_lines("events.jsonl"))))

for record in pipeline:
    print(record)

Each stage is small, testable, and streams. No intermediate list ever holds the full dataset.

6. `itertools` — every pipeline helper you need

python

from itertools import (
    chain, zip_longest, accumulate, count, cycle, islice,
    groupby, combinations, permutations, product, takewhile,
    dropwhile, tee, pairwise, starmap, batched,    # batched in 3.12+
)

Highlights:

chain(*its): flatten one level.
zip_longest(a, b, fillvalue=None): zip until the longer is exhausted.
accumulate(it, func=op.add): running totals.
count(start, step): infinite counter.
cycle(it): repeat forever.
islice(it, start, stop, step): slice a stream.
groupby(it, key): groups of consecutive equal-key items.
combinations(it, r), permutations(it, r): math.
product(*its): Cartesian product.
takewhile(pred, it), dropwhile(pred, it): prefix/suffix.
tee(it, n): split an iterator into n independent ones.
pairwise(it): yields overlapping pairs (3.10+).
batched(it, n): chunks of n (3.12+) — bye-bye chunked utility functions.

`batched` for batched API calls

python

from itertools import batched
for chunk in batched(ids, 100):
    api.bulk_lookup(list(chunk))

7. Generator state: `send`, `throw`, `close`

Generators are coroutines in disguise. yield is also an expression that can receive a value from outside.

python

def echo():
    while True:
        x = yield               # receives from .send()
        print("got", x)

g = echo()
next(g)                         # advance to first yield
g.send("hi")                    # got hi
g.send("ho")                    # got ho
g.close()                       # raises GeneratorExit inside, ends

g.throw(SomeException) injects an exception at the yield point — useful for graceful shutdown.

This is the foundation of asyncio (Phase 4.4). Modern code uses async def and await instead of send/throw, but knowing the mechanism helps debug.

8. Generator vs iterator class — when to write a class

Need	Tool
Stateful one-shot iteration	Generator function
Reusable iteration over an instance	Class with `__iter__` returning a fresh generator
Random access + iteration	Class with `__getitem__` + `__len__`

python

class Sentence:
    def __init__(self, text):
        self.words = text.split()
    def __iter__(self):
        yield from self.words                  # fresh iterator each call

s = Sentence("hello world")
list(s); list(s)                                # both work, independent

If the class only needs to be iterable, this is the cleanest pattern.

9. `for ... else` on iterators (and the `break` rule)

python

def find(items, predicate):
    for x in items:
        if predicate(x):
            return x
    return None

# or:
def find(items, predicate):
    for x in items:
        if predicate(x):
            break
    else:
        x = None
    return x

The else clause runs only if the loop completes without break. Useful with iterators; rare in practice.

10. Closing generators properly

Inside a generator with try/finally:

python

def db_rows():
    conn = open_db()
    try:
        for row in conn.query("SELECT *"):
            yield row
    finally:
        conn.close()

When the consumer stops early (break), Python calls gen.close(), which fires GeneratorExit at the yield. The finally block runs and closes the connection.

This makes generators safe for resource management — better than callback-based APIs.

11. Asynchronous generators (preview)

python

async def async_lines(url):
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", url) as r:
            async for line in r.aiter_lines():
                yield line

async def main():
    async for line in async_lines("https://example.com/log"):
        print(line)

async def with yield → async generator. Consumed with async for. Phase 4.4 covers asyncio in depth.

12. Performance notes

Generators are slightly slower per-item than list comprehensions (due to overhead per yield) but use far less memory.
For tight numeric loops, NumPy / vectorisation beat generators every time.
If profiling shows a generator hot, consider converting to a list or rewriting in itertools / NumPy.

Hands-on lab (2 hours)

Write fibonacci() as an infinite generator; list(islice(fibonacci(), 10)) for first 10.
Write read_csv_streaming(path) that yields dicts one row at a time using csv.DictReader.
Build a pipeline read → parse JSON → filter ERROR → batch 100 → POST to API.
Implement pairwise(it) from scratch (then check against itertools.pairwise).
Use itertools.groupby to compute per-day counts from a sorted JSONL log.
Build a class Tree with a __iter__ that does a depth-first traversal using yield from.
Write a generator with try/finally that closes a resource; verify cleanup when consumer breaks early.

Common pitfalls

Re-using an exhausted iterator (it's empty; convert to list once if needed).
Returning a generator from a function whose caller expects a list — they len() it and get TypeError.
Side effects in a generator expression ((do_thing(x) for x in xs) won't run until iterated).
Forgetting that a generator function returning early raises StopIteration with a value (rarely useful; common confusion).
yield from g vs return g: very different.

Self-check

Difference between iterable and iterator.
What does yield from do?
When use a generator vs list comprehension?
What is itertools.batched for?
How does the GeneratorExit mechanism support cleanup?

References

Fluent Python, Ramalho — Chapter 17.
PEP 255 — Simple Generators.
PEP 380 — Syntax for Delegating to a Subgenerator.
Effective Python, Slatkin — Items 30-43.
David Beazley, "Generators: The Final Frontier" (PyCon talk).

Why this matters

Learning objectives

1. The iterator protocol

Iterable vs Iterator

2. Generator functions — yield

Reading from a file lazily

3. Generator expressions

4. yield from — composition

Flattening

5. Pipelines

6. itertools — every pipeline helper you need

batched for batched API calls

7. Generator state: send, throw, close

8. Generator vs iterator class — when to write a class

9. for ... else on iterators (and the break rule)

10. Closing generators properly

11. Asynchronous generators (preview)

12. Performance notes

Hands-on lab (2 hours)

Common pitfalls

Self-check

References

2. Generator functions — `yield`

4. `yield from` — composition

6. `itertools` — every pipeline helper you need

`batched` for batched API calls

7. Generator state: `send`, `throw`, `close`

9. `for ... else` on iterators (and the `break` rule)