Lazy by Design: A Practical Deep Dive into Python Generators and yield

Most Python code builds a list, fills it up, and then loops over it. That works fine until the data gets big — a multi-gigabyte log file, an API that paginates forever, or a stream you can't hold in memory all at once. This is exactly where generators shine. A generator computes its values lazily, one at a time, only when you ask for the next one. That single idea unlocks constant-memory processing, infinite sequences, and elegant data pipelines.

In this deep dive we'll build up from the basics of yield to generator expressions, yield from, two-way communication with send(), and the pitfalls that surprise people. Every example is short and runnable.

What a generator actually is

Any function that contains the yield keyword becomes a generator function. Calling it doesn't run the body — it hands you a generator object. The body only advances when you iterate, and it pauses at each yield, remembering exactly where it left off.

def countdown(n):
    while n > 0:
        yield n
        n -= 1

gen = countdown(3)
print(next(gen))   # 3
print(next(gen))   # 2
print(list(gen))   # [1]  -- continues from where it paused

Each call to next() runs until the next yield, returns that value, and freezes the function's state — local variables, the instruction pointer, everything. When the function finally returns (or falls off the end), Python raises StopIteration, which is what ends a for loop cleanly.

Why laziness matters: memory

A list comprehension materializes every element up front. A generator expression — same syntax but with parentheses — produces values on demand and stays a fixed, tiny size no matter how many elements it will yield.

import sys

list_comp = [x * x for x in range(1_000_000)]   # builds a million ints now
gen_expr  = (x * x for x in range(1_000_000))   # builds nothing yet

print(type(gen_expr))              # <class 'generator'>
print(sys.getsizeof(gen_expr))     # a few hundred bytes, regardless of range

The generator's size doesn't grow with the data because it never holds the data — it holds the recipe. When a consumer like sum() can accept an iterator, you never build the intermediate list at all:

# No temporary million-element list is created:
total = sum(x * x for x in range(1_000_000))

Infinite and streaming sequences

Because values are produced on demand, a generator can represent an infinite sequence. You just need something to stop pulling — like itertools.islice.

import itertools

def integers():
    n = 0
    while True:
        yield n
        n += 1

first_five = list(itertools.islice(integers(), 5))
print(first_five)   # [0, 1, 2, 3, 4]

The same pattern is perfect for reading large files. Iterating a file object already yields lines lazily; wrapping it in a generator lets you filter and transform records without ever loading the whole file:

def read_records(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:                 # skip blanks
                yield line.upper()

# Processes a 10 GB file in constant memory:
for record in read_records("huge.log"):
    ...

Composing pipelines

Generators chain naturally. Each stage pulls one item from the stage before it, so data flows through the whole pipeline one element at a time — no stage ever needs the full dataset.

def evens(source):
    for n in source:
        if n % 2 == 0:
            yield n

def squared(source):
    for n in source:
        yield n * n

pipeline = squared(evens(range(10)))
print(list(pipeline))   # [0, 4, 16, 36, 64]

Read that inside-out: range(10) feeds evens, which feeds squared. Swap or add stages freely; the memory profile stays flat.

Delegating with yield from

When one generator needs to yield everything from another iterable, yield from replaces a manual loop and also forwards send() and exceptions to the sub-generator.

def chain_all(*iterables):
    for it in iterables:
        yield from it

print(list(chain_all([1, 2], (3, 4), "xy")))
# [1, 2, 3, 4, 'x', 'y']

yield from is especially handy for recursive structures, like flattening nested lists, where each recursive call delegates its output upward.

Two-way communication: send()

Generators aren't just producers — you can push values into them. The yield expression evaluates to whatever you pass to send(), which turns a generator into a lightweight coroutine. You must "prime" it with one next() call first to reach the initial yield.

def running_average():
    total, count, average = 0.0, 0, None
    while True:
        value = yield average       # receive via send(), emit current average
        total += value
        count += 1
        average = total / count

avg = running_average()
next(avg)                # prime it
print(avg.send(10))      # 10.0
print(avg.send(20))      # 15.0
print(avg.send(30))      # 20.0

Return values and cleanup

A generator can return a value. It won't appear during iteration; instead it's attached to the StopIteration exception, which yield from also surfaces as the delegation's result.

def with_return():
    yield 1
    yield 2
    return "done"

g = with_return()
try:
    while True:
        next(g)
except StopIteration as e:
    print(e.value)   # done

If a generator holds a resource, wrap the loop in try/finally. Calling close() (or garbage-collecting the generator) raises GeneratorExit at the paused yield, so your finally block runs.

def managed():
    try:
        while True:
            yield "working"
    finally:
        print("cleanup ran")

m = managed()
print(next(m))   # working
m.close()        # cleanup ran

Common pitfalls

Generators are single-use. Once exhausted, they yield nothing more. Iterating twice silently gives you an empty result the second time:

g = (x for x in range(3))
print(list(g))   # [0, 1, 2]
print(list(g))   # []  -- already consumed

If you need to iterate more than once, either recreate the generator or materialize it into a list. Related traps worth remembering: len() doesn't work on a generator (it has no known length), indexing and slicing with [] don't work (use itertools.islice), and because the body is lazy, exceptions raised inside it only fire when you actually iterate — not when you call the function. Finally, watch out for late binding when building generators in a loop; capture loop variables explicitly if the generator is consumed later.

Wrap-up and next steps

Generators give you a simple mental model with outsized benefits: pause-and-resume execution via yield, constant memory over arbitrarily large or infinite data, composable pipelines, and optional two-way communication through send(). Reach for a generator expression whenever you're about to build a list only to loop over it once, and write a generator function whenever you're streaming or transforming records.

From here, explore the itertools module for battle-tested building blocks like chain, groupby, and tee, and look at how async def with yield extends these ideas into asynchronous generators. The lazy mindset you build with yield will change how you approach data processing in Python.