Python Generators Explained: yield, Lazy Pipelines, and Memory-Efficient Code

Most Python developers meet yield early, nod politely, and go back to building lists. That's a shame, because generators are one of the language's most practical features: they let you process data one item at a time, on demand, without ever holding the whole dataset in memory. They're the reason you can iterate over a 10 GB log file on a laptop with 8 GB of RAM, and they're the foundation that itertools, asyncio, and countless libraries are built on.

In this post we'll build a solid mental model of how generators actually work, then use them to construct lazy data pipelines you can drop into real projects.

The problem: eager code holds everything in memory

Consider a function that reads numbers from a file and returns the squares of the even ones:

def even_squares(path):
    results = []
    with open(path) as f:
        for line in f:
            n = int(line)
            if n % 2 == 0:
                results.append(n * n)
    return results

This works, but it's eager: the entire result list exists in memory before the caller sees a single value. For a million lines that's a million-element list — even if the caller only wants the first ten results.

Enter yield

Replace append with yield and delete the list:

def even_squares(path):
    with open(path) as f:
        for line in f:
            n = int(line)
            if n % 2 == 0:
                yield n * n

The presence of yield anywhere in a function body changes what the function is. Calling it no longer runs the body — it returns a generator object:

>>> gen = even_squares("numbers.txt")
>>> gen
<generator object even_squares at 0x7f3a2c1e4dd0>

Nothing has executed yet — the file isn't even open. Code runs only when you ask for a value:

>>> next(gen)
4
>>> next(gen)
16

Each next() call runs the body until it hits a yield, hands you that value, and freezes. All local variables, the file position, the loop state — everything is preserved exactly where it stopped. The next call resumes from that frozen point. When the function body finally ends, the generator raises StopIteration, which is how for loops know to stop.

This suspend-and-resume behavior is the whole trick. A generator is a function that can pause.

Generators are iterators

Generator objects implement the iterator protocol (__iter__ and __next__), so anything that accepts an iterable accepts a generator:

total = sum(even_squares("numbers.txt"))

for sq in even_squares("numbers.txt"):
    print(sq)

first_ten = list(itertools.islice(even_squares("numbers.txt"), 10))

One important consequence: generators are single-use. Once exhausted, they stay exhausted:

>>> gen = (x * x for x in range(3))
>>> list(gen)
[0, 1, 4]
>>> list(gen)   # already consumed
[]

If you need to iterate twice, call the generator function again to get a fresh generator — or rethink whether you actually need two passes.

Generator expressions: comprehensions without the list

A generator expression looks like a list comprehension with parentheses instead of brackets:

squares_list = [x * x for x in range(1_000_000)]   # ~40 MB, built now
squares_gen  = (x * x for x in range(1_000_000))   # ~200 bytes, built lazily

Use a genexp whenever the result is consumed once and immediately — especially as an argument to functions like sum, max, any, or all. As a bonus, when a genexp is the sole argument you can drop the extra parentheses:

total = sum(x * x for x in range(1_000_000))
has_negatives = any(n < 0 for n in numbers)

any and all short-circuit, so paired with a genexp they stop consuming input the moment the answer is known — an eager list would have done all the work up front for nothing.

Building lazy pipelines

Generators compose beautifully. Each stage consumes the previous one, and no stage does any work until the final consumer pulls a value:

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def parse_records(lines):
    for line in lines:
        if line and not line.startswith("#"):
            yield line.split(",")

def to_floats(records, column):
    for rec in records:
        try:
            yield float(rec[column])
        except (ValueError, IndexError):
            continue  # skip malformed rows

# Wire the stages together — still no work done:
lines   = read_lines("measurements.csv")
records = parse_records(lines)
values  = to_floats(records, column=2)

# Work happens here, one row at a time:
print(max(values))

This pipeline processes a file of any size in constant memory. Each line flows through all three stages before the next line is read. The structure also keeps concerns separated: reading, parsing, and conversion are independent, individually testable functions.

yield from: delegating to sub-generators

yield from yields every item of another iterable, which is handy for flattening and for splitting big generators into helpers:

def walk(node):
    yield node.value
    for child in node.children:
        yield from walk(child)   # recursive traversal, still lazy

Without yield from you'd need an inner for loop at every delegation point; with it, recursive tree walks read almost like the eager version.

Common pitfalls

1. Forgetting that nothing runs until iteration

def load(path):
    f = open(path)   # ← does NOT run at call time
    ...
    yield ...

Validation and error handling inside a generator only fire on the first next(). If you want an immediate error for a missing file, validate eagerly in a plain function that returns a generator, or open the file before the first yield runs by splitting the function.

2. Consuming a generator you still need

values = (float(x) for x in raw)
maximum = max(values)
minimum = min(values)   # ValueError: empty — already exhausted!

Either materialize once with list() when you need multiple passes, or compute both aggregates in a single pass.

3. Leaking resources on partial consumption

If a consumer abandons a generator mid-flight, cleanup in a with block inside the generator runs only when the generator object is garbage-collected or explicitly closed. When timing matters, close it deterministically:

gen = read_lines("huge.log")
try:
    for line in gen:
        if "ERROR" in line:
            break
finally:
    gen.close()   # runs the generator's finally/with cleanup now

Or wrap it with contextlib.closing:

from contextlib import closing

with closing(read_lines("huge.log")) as gen:
    for line in gen:
        ...

When not to use generators

Generators aren't free wins everywhere. Reach for a plain list when the dataset is small and you need random access, len(), slicing, or multiple passes; when you want errors raised at call time rather than first-iteration time; or when you need to serialize the result. Laziness is a tool for streams, not a default for every function.

Wrap-up and next steps

Generators turn functions into pausable, resumable producers of values. That single idea gives you constant-memory processing of arbitrarily large data, composable pipelines, and short-circuiting aggregations — all with less code than the eager equivalents. Start by converting one list-building loop in your codebase into a yield-based generator and see how the call sites simplify.

From here, two natural follow-ups: the itertools module, which provides battle-tested building blocks (islice, chain, takewhile) for generator pipelines, and asyncio, whose coroutines grew directly out of the generator machinery described here. We've covered both in earlier deep dives on this blog.