Lazy, Stateful, and Infinite: A Practical Deep Dive into Python Generators and yield

Most Python programmers meet generators the same way: they see a function with yield in it, notice it "returns something you can loop over," and move on. That surface understanding gets you surprisingly far — but it also hides the reason generators exist. Generators are Python's built-in tool for lazy evaluation: computing values one at a time, only when asked, without ever holding the whole sequence in memory. That single idea unlocks infinite streams, memory-flat data pipelines, and a clean way to write iterators without ceremony.

This guide takes generators apart and puts them back together. We'll start with the memory problem they solve, walk through how yield actually suspends a function, and finish with the advanced features — send(), yield from, and proper cleanup — that turn generators into more than fancy loops.

The problem: building a list you don't need all at once

Suppose you want the squares of the first million integers. The obvious approach builds a list:

def squares_list(n):
    return [x * x for x in range(n)]

data = squares_list(1_000_000)  # allocates ~8 MB up front

That works, but you've paid for a million integers in memory before touching a single one. A generator computes them on demand instead:

def squares_gen(n):
    for x in range(n):
        yield x * x

g = squares_gen(1_000_000)

The difference is dramatic. The list of a million squares occupies megabytes; the generator object is a fixed, tiny handle regardless of n:

import sys

print(sys.getsizeof(squares_gen(1_000_000)))  # 104 (bytes) — constant

The generator hasn't computed anything yet. It's a paused recipe. Values appear only as you iterate, and each one is discarded once you move on unless you keep a reference to it.

How yield actually works

A regular function runs top to bottom and returns once. A function containing yield is different: calling it doesn't run the body at all — it returns a generator object. The body executes only when you call next() on that object (which is what a for loop does under the hood). Execution runs until it hits a yield, hands back that value, and freezes the function's entire state — local variables, the instruction pointer, everything. The next next() thaws it and resumes right after the yield.

def demo():
    print("start")
    yield 1
    print("resumed after first yield")
    yield 2
    print("about to finish")

g = demo()
print(next(g))   # prints "start", then 1
print(next(g))   # prints "resumed after first yield", then 2
next(g)          # prints "about to finish", then raises StopIteration

When the function body finally returns (or falls off the end), Python raises StopIteration to signal the sequence is over. A for loop catches that exception silently, which is why you never see it in normal use. This freeze-and-resume behavior is the whole trick: a generator is a function that remembers where it left off.

Generator expressions: comprehension syntax, lazy semantics

You don't always need a full function. A generator expression looks like a list comprehension but uses parentheses, and it produces values lazily:

# List comprehension — builds the whole list
squares = [x * x for x in range(1000)]

# Generator expression — produces values on demand
squares = (x * x for x in range(1000))

# When the generator is the sole argument, you can drop the extra parens
total = sum(x * x for x in range(1000))
print(total)  # 332833500

Reach for a generator expression whenever you're feeding a sequence straight into a consumer like sum(), any(), max(), or "".join(...). You get the readability of a comprehension without materializing an intermediate list.

Infinite streams, made safe

Because generators produce values only on request, they can represent sequences with no end. Here's a counter that runs forever:

def count_from(start):
    n = start
    while True:
        yield n
        n += 1

Calling list(count_from(0)) would hang forever, so you bound the stream at the consumption end. itertools.islice is the idiomatic way to take a finite slice:

from itertools import islice

c = count_from(100)
print(list(islice(c, 5)))  # [100, 101, 102, 103, 104]

This separation of concerns — the generator describes what the sequence is, the consumer decides how much of it to use — is one of the most powerful patterns generators enable.

Building data pipelines

Generators compose beautifully. Each stage takes an iterable and yields a transformed iterable, so you can chain filters and maps into a pipeline where data flows one item at a time — no stage ever buffers the whole dataset.

def read_lines(text):
    for line in text.splitlines():
        yield line

def grep(pattern, lines):
    for line in lines:
        if pattern in line:
            yield line

def upper(lines):
    for line in lines:
        yield line.upper()

sample = "apple pie\nbanana bread\napple tart\ncherry cake"
pipeline = upper(grep("apple", read_lines(sample)))
print(list(pipeline))  # ['APPLE PIE', 'APPLE TART']

The same shape scales to real work — streaming a multi-gigabyte log file through filters without loading it into RAM. The classic idiom for lazy file processing is a generator that iterates the file object, which itself yields lines one at a time:

def read_file(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

# Count matching lines in a huge file with flat memory use
errors = sum(1 for line in read_file("app.log") if "ERROR" in line)

Two-way communication with send()

Generators aren't only a source of values — they can also receive them. When you write value = yield x, the yield expression evaluates to whatever the caller passes via gen.send(...). This turns a generator into a small stateful coroutine. Here's a running average that remembers everything it has seen:

def running_average():
    total = 0.0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

avg = running_average()
next(avg)              # "prime" the generator: run up to the first yield
print(avg.send(10))    # 10.0
print(avg.send(20))    # 15.0
print(avg.send(30))    # 20.0

Note the priming step. A freshly created generator is paused before the first line, so you must advance it to the first yield with next() (or send(None)) before you can send real values. Forgetting to prime is the single most common send() bug.

Delegating with yield from

When one generator needs to yield everything from another iterable, the manual version is a loop:

def chain(*iterables):
    for it in iterables:
        for item in it:
            yield item

Since Python 3.3, yield from expresses this directly and does more than save two lines — it transparently forwards send() values and exceptions to the sub-generator and captures its return value:

def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain([1, 2], (3, 4), "ab")))  # [1, 2, 3, 4, 'a', 'b']

A generator can even return a value, which yield from surfaces as the result of the delegation expression:

def worker():
    yield 1
    yield 2
    return "done"

def driver():
    result = yield from worker()
    print("worker returned:", result)  # worker returned: done

list(driver())

Cleanup, exhaustion, and common pitfalls

A few sharp edges are worth internalizing.

Generators are exhausted after one pass. Once you've iterated a generator to the end, it's spent — iterating again yields nothing. This trips people up when they store a generator and try to loop over it twice:

def gen():
    yield 1
    yield 2

g = gen()
print(list(g))  # [1, 2]
print(list(g))  # [] — already exhausted

If you need to iterate more than once, either recreate the generator or materialize it into a list first.

Use try/finally for cleanup. If a generator holds a resource, wrap the yielding logic in try/finally. When the generator is garbage-collected or you call .close() on it, Python throws GeneratorExit at the paused yield, and your finally block runs:

def managed():
    try:
        while True:
            yield "working"
    finally:
        print("generator cleaned up")

m = managed()
print(next(m))  # working
m.close()       # prints "generator cleaned up"

Don't confuse laziness with free repetition. Because nothing is computed until consumed, side effects in a generator only fire when you actually iterate. That's usually what you want, but it means a generator you never consume does no work at all — occasionally a source of "why didn't my code run?" confusion.

Wrap-up and next steps

Generators give you three things at once: lazy evaluation that keeps memory flat, the ability to model infinite or streaming sequences, and a concise way to write iterators and composable pipelines. Start simple — swap a list comprehension for a generator expression when you're piping data into sum() or a loop, and write generator functions for any sequence you produce incrementally. Once that feels natural, layer in yield from for delegation and send() when you need a stateful coroutine.

From here, explore the itertools module, which is a toolbox of ready-made generators (islice, chain, groupby, takewhile) designed to snap together with your own. And when you're ready to make laziness concurrent, Python's async def and async for build directly on the same suspend-and-resume machinery you now understand.