Lazy by Design: A Practical Deep Dive into Python's itertools

Why itertools deserves a permanent spot in your toolbox

Most Python code loops. We reach for for, we build lists, we filter, we group, we paginate. And most of the time we do it by hand — accumulating into a fresh list, tracking an index, slicing, nesting comprehensions until they stop being readable. The itertools module exists to replace those hand-rolled loops with a small vocabulary of composable, lazy building blocks. Every function in it returns an iterator: nothing is computed until you ask for the next value, and nothing is stored that you don't keep. That means you can express pipelines over millions of rows — or even infinite streams — in constant memory.

The module is written in C, it has been stable for years, and it ships with every Python install. Learning it well pays off forever. This guide walks through the three families of tools it provides, then shows how to combine them into real pipelines and how to avoid the handful of traps that catch newcomers.

The three families

The functions in itertools fall into three natural groups: infinite iterators that generate values forever, terminating iterators that transform or consume a finite input, and combinatoric iterators that enumerate products, permutations, and combinations. We'll take them in that order.

Infinite iterators: count, cycle, repeat

These never stop on their own, so you always pair them with something that limits them — usually islice, zip, or a break.

import itertools as it

# count(start, step) — a lazy replacement for a manual counter
print(list(it.islice(it.count(10, 2), 5)))
# [10, 12, 14, 16, 18]

# cycle — loop over a sequence forever (great for round-robin assignment)
colors = it.cycle(["red", "green", "blue"])
print([next(colors) for _ in range(5)])
# ['red', 'green', 'blue', 'red', 'green']

# repeat — the same value, optionally a fixed number of times
print(list(map(pow, range(5), it.repeat(2))))
# [0, 1, 4, 9, 16]  (each base raised to the power 2)

count is the idiomatic way to attach an incrementing number to a stream without a manual variable, and repeat shines as a constant argument to map — you avoid writing a lambda just to supply a fixed second parameter.

Terminating iterators: the everyday workhorses

This is the group you will use most. chain flattens several iterables into one stream; its from_iterable variant flattens a single iterable of iterables (think "list of lists").

print(list(it.chain("AB", [1, 2], (True,))))
# ['A', 'B', 1, 2, True]

rows = [[1, 2], [3, 4], [5, 6]]
print(list(it.chain.from_iterable(rows)))
# [1, 2, 3, 4, 5, 6]

accumulate produces running results. By default it's a running sum, but pass any binary function to get running products, running maxima, or anything else.

import operator

print(list(it.accumulate([1, 2, 3, 4, 5])))              # [1, 3, 6, 10, 15]
print(list(it.accumulate([1, 2, 3, 4, 5], operator.mul))) # [1, 2, 6, 24, 120]
print(list(it.accumulate([3, 1, 4, 1, 5, 9, 2], max)))    # [3, 3, 4, 4, 5, 9, 9]

compress, takewhile, dropwhile, and filterfalse are a family of selectors. compress keeps items where a parallel selector is truthy; takewhile yields until the predicate first fails; dropwhile skips until it first fails and then yields everything after; filterfalse is filter inverted.

print(list(it.compress("ABCDEF", [1, 0, 1, 0, 1, 1])))  # ['A', 'C', 'E', 'F']
print(list(it.takewhile(lambda x: x < 5, [1, 4, 6, 2]))) # [1, 4]
print(list(it.dropwhile(lambda x: x < 5, [1, 4, 6, 2]))) # [6, 2]
print(list(it.filterfalse(lambda x: x % 2, range(10))))  # [0, 2, 4, 6, 8]

starmap is map for pre-zipped argument tuples, and zip_longest pads the short iterables instead of stopping at the shortest.

print(list(it.starmap(pow, [(2, 3), (3, 2)])))  # [8, 9]
print(list(it.zip_longest("AB", "wxyz", fillvalue="-")))
# [('A', 'w'), ('B', 'x'), ('-', 'y'), ('-', 'z')]

Two newer additions are worth knowing. pairwise (Python 3.10+) yields overlapping adjacent pairs — perfect for computing differences between consecutive elements. batched (Python 3.12+) chops an iterable into fixed-size tuples, ideal for paging API calls or bulk database inserts.

print(list(it.pairwise("ABCDE")))
# [('A', 'B'), ('B', 'C'), ('C', 'D'), ('D', 'E')]

# Python 3.12+
# print(list(it.batched("ABCDEFG", 3)))
# [('A', 'B', 'C'), ('D', 'E', 'F'), ('G',)]

If you're on an older Python, batched is a three-line recipe (shown near the end).

groupby: powerful, but read the fine print

groupby collapses consecutive items that share a key into groups. The word "consecutive" is the whole story: it does not sort for you, so you almost always sort by the same key first.

data = [("a", 1), ("a", 2), ("b", 3), ("a", 4)]
data.sort(key=lambda p: p[0])            # sort FIRST — required

for key, group in it.groupby(data, key=lambda p: p[0]):
    print(key, [value for _, value in group])
# a [1, 2, 4]
# b [3]

Skip the sort and you'd get two separate a groups, because the first two a rows and the later a row aren't adjacent. There's a second subtlety: each group is itself a shared iterator that's consumed as you advance to the next group, so materialize it (with list or a comprehension) inside the loop if you need it later.

Combinatoric iterators

These enumerate the classic combinatorics without you writing nested loops. product is the Cartesian product (a stand-in for nested for loops), permutations respects order, combinations ignores it, and the _with_replacement variant allows repeats.

print(list(it.product([0, 1], repeat=2)))
# [(0, 0), (0, 1), (1, 0), (1, 1)]

print(list(it.permutations("ABC", 2)))
# [('A','B'), ('A','C'), ('B','A'), ('B','C'), ('C','A'), ('C','B')]

print(list(it.combinations("ABCD", 2)))
# [('A','B'), ('A','C'), ('A','D'), ('B','C'), ('B','D'), ('C','D')]

print(list(it.combinations_with_replacement("AB", 2)))
# [('A','A'), ('A','B'), ('B','B')]

A word of caution: these grow explosively. permutations of ten items is over three million tuples. Because the results are lazy you can iterate them one at a time without building the whole list, but if you wrap them in list() you materialize everything — so only do that for small inputs.

Composing pipelines

The real power appears when you chain these tools. Because each returns an iterator, stacking them builds a single lazy pipeline that reads one item at a time from source to sink. Here's a running monthly total, computed without an accumulator variable:

rows = [("jan", 100), ("feb", 150), ("mar", 120)]
running = it.accumulate(amount for _, amount in rows)
print(list(zip((month for month, _ in rows), running)))
# [('jan', 100), ('feb', 250), ('mar', 370)]

And a sliding window over any iterable — a common need for smoothing or n-gram analysis — built from islice:

def sliding_window(iterable, n):
    iterator = iter(iterable)
    window = tuple(it.islice(iterator, n))
    if len(window) == n:
        yield window
    for item in iterator:
        window = window[1:] + (item,)
        yield window

print(list(sliding_window("ABCDEF", 3)))
# [('A','B','C'), ('B','C','D'), ('C','D','E'), ('D','E','F')]

The itertools recipes

The official documentation includes a "recipes" section of small functions built from the primitives. Two are so useful they're worth memorizing. unique_everseen deduplicates while preserving order, and a hand-rolled batched works on any Python version.

def unique_everseen(iterable, key=None):
    seen = set()
    for element in iterable:
        k = element if key is None else key(element)
        if k not in seen:
            seen.add(k)
            yield element

print(list(unique_everseen("AAAABBBCCDAABBB")))  # ['A', 'B', 'C', 'D']

def batched(iterable, n):
    if n < 1:
        raise ValueError("n must be at least one")
    iterator = iter(iterable)
    while batch := tuple(it.islice(iterator, n)):
        yield batch

print(list(batched(range(10), 4)))
# [(0, 1, 2, 3), (4, 5, 6, 7), (8, 9)]

Common pitfalls

Iterators are exhausted once. After you consume an itertools result — even by printing it — it's empty. If you need to walk the same stream twice, either convert it to a list or duplicate it with tee. But note that tee buffers everything one branch has seen that the other hasn't, so it's cheap only when the branches advance together and useless as a replacement for re-iterating an expensive source.

a, b = it.tee([1, 2, 3])
print(list(a), list(b))  # [1, 2, 3] [1, 2, 3]

Infinite iterators need a brake. Calling list(it.count()) will hang and eventually run you out of memory. Always cap them with islice, zip against a finite iterable, or a break.

groupby needs a prior sort unless the data is already grouped, and its groups are consumed lazily — materialize each group inside the loop if you'll use it after moving on.

Combinatoric explosions are silent. The functions are lazy, so the danger only appears when you collect the results. Iterate them, filter them early, and avoid wrapping large inputs in list().

Wrap-up and next steps

itertools rewards a small upfront investment with cleaner, faster, more memory-efficient code across your entire career with Python. Start by replacing three habits: swap manual counters for count, swap "list of lists" flattening for chain.from_iterable, and swap hand-written running totals for accumulate. From there, reach for groupby whenever you're bucketing sorted data and batched whenever you're chunking work.

The best next step is to read the module's own documentation, especially the recipes section — it's a compact masterclass in composing iterators. Then, next time you catch yourself building a list inside a loop just to iterate it once, ask whether a lazy pipeline would say it more clearly. More often than not, it will.