Find the Slow Parts First: Profiling Python with timeit, cProfile, and tracemalloc

Every developer has a story about spending an afternoon optimizing a function that turned out to account for 2% of the runtime. Intuition about performance is notoriously unreliable — the slow part is almost never where you think it is. The fix is cheap: Python ships everything you need to measure instead of guess. This post walks through the three standard-library tools that cover most performance work — timeit for micro-benchmarks, cProfile for finding hot functions, and tracemalloc for memory — plus the mistakes that make measurements lie to you.

Quick timing: use perf_counter, not time.time

For a rough "how long does this block take", wrap it with time.perf_counter():

from time import perf_counter

start = perf_counter()
result = process_records(records)
elapsed = perf_counter() - start
print(f"process_records took {elapsed:.3f}s")

perf_counter() uses the highest-resolution clock available and is monotonic — it never jumps backwards. time.time() is wall-clock time and can shift with NTP adjustments or DST, which makes it the wrong tool for measuring durations. This pattern is fine for coarse timing, but for anything that runs in microseconds a single measurement is mostly noise. That's what timeit is for.

Micro-benchmarks with timeit

The timeit module runs a snippet many times in a clean environment and reports the total, averaging out per-run noise. The fastest way to use it is the command line:

$ python -m timeit "'-'.join(str(n) for n in range(100))"
20000 loops, best of 5: 12.1 usec per loop

$ python -m timeit "'-'.join([str(n) for n in range(100)])"
20000 loops, best of 5: 10.8 usec per loop

(Numbers vary by machine — run them yourself.) The CLI picks a sensible loop count automatically and reports the best of five runs. From code, use timeit.repeat() and take the minimum:

import timeit

setup = """
import random
random.seed(1)
data = [random.random() for _ in range(10_000)]
"""

times = timeit.repeat("sorted(data)", setup=setup, repeat=5, number=100)
print(f"best: {min(times) / 100 * 1e6:.1f} µs per call")

Why the minimum and not the average? Slower runs are slower because of interference — garbage collection, other processes, CPU frequency scaling. The minimum is the closest measurement to what your code actually costs. Two more details: put data construction in setup so you don't time it, and seed randomness so runs are comparable.

Micro-benchmark pitfalls

Micro-benchmarks answer narrow questions ("is a list comprehension faster than a generator here?"). They do not tell you whether that difference matters in your program — a 2 µs win inside a function called ten times is nothing. Never sacrifice readability based on a micro-benchmark alone; first confirm the code is actually hot, which brings us to profiling.

Finding hot functions with cProfile

cProfile instruments every function call and tells you where the time went. The one-liner version profiles a whole script:

$ python -m cProfile -s cumulative my_script.py

The -s cumulative flag sorts by cumulative time — the time spent in a function including everything it calls — which is usually the right lens for finding the expensive call paths. For targeted profiling inside a program, cProfile.Profile works as a context manager (Python 3.8+):

import cProfile
import pstats

with cProfile.Profile() as profiler:
    run_pipeline()

stats = pstats.Stats(profiler)
stats.sort_stats("cumulative")
stats.print_stats(10)          # top 10 entries only

Reading the output: ncalls is how often the function ran, tottime is time spent in the function itself (excluding callees), and cumtime includes callees. A function with high cumtime but low tottime is just a caller — follow the chain down to the function where tottime is actually large. A function with a huge ncalls is often the real story: the fix is calling it less, not making it faster.

Two caveats. First, cProfile adds real overhead to every function call, so absolute numbers are inflated — use it to find where time goes, then verify improvements with timeit or perf_counter without the profiler attached. Second, it works at function granularity; it won't tell you which line inside a long function is slow.

Memory profiling with tracemalloc

Slow is not the only problem — sometimes the issue is a process that balloons to gigabytes. tracemalloc records every allocation Python makes, with the source line that made it:

import tracemalloc

tracemalloc.start()

data = load_everything()       # the suspect code

snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno")[:5]:
    print(stat)
# e.g. loader.py:42: size=210 MiB, count=1500000, average=147 B

Even more useful is comparing two snapshots to see what a specific operation allocated:

before = tracemalloc.take_snapshot()
cache.warm_up()
after = tracemalloc.take_snapshot()

for stat in after.compare_to(before, "lineno")[:5]:
    print(stat)

Start tracing as early as possible — allocations made before tracemalloc.start() are invisible to it. And expect a noticeable slowdown and extra memory use while tracing is on; it's a diagnostic tool, not something to leave enabled in production.

A practical workflow

The tools slot into a simple loop. Establish a baseline first: one number (seconds, MB) for the realistic workload you care about, measured with perf_counter or your test suite. Then profile with cProfile to find the top two or three functions by tottime — resist touching anything else. Fix one thing: often an algorithmic change (a set lookup instead of a list scan, batching database calls, caching a repeated computation) beats any amount of line-level tweaking. Re-measure against the baseline, and keep the numbers in the commit message so the next person knows what was gained. Repeat until the program is fast enough — and stop there. Code clarity has a longer shelf life than a microsecond.

The classic mistakes to avoid: benchmarking with unrealistically small data (many costs only show at scale), timing the first run of something that caches (warm up first, or measure both cold and warm deliberately), comparing runs across different machines or Python versions, and trusting profiler-inflated absolute numbers instead of before/after ratios.

When the standard library isn't enough

Three third-party tools are worth knowing once you outgrow the stdlib. line_profiler gives per-line timings inside functions you decorate — the answer to cProfile's function-level granularity. py-spy is a sampling profiler that attaches to a running process without modifying it and adds almost no overhead, which makes it safe for production debugging. And scalene profiles CPU, memory, and even GPU together with line-level detail. All are a pip install away, but reach for them only after the stdlib tools have taken you as far as they can — which is usually further than you'd expect.

Wrap-up and next steps

Performance work is measurement work. Use perf_counter for coarse timing, timeit (taking the minimum of repeats) for micro-benchmarks, cProfile with pstats to find the functions that actually matter, and tracemalloc snapshots when memory is the problem. Profile before optimizing, verify after, and let the numbers overrule your intuition — they will, regularly. A good next exercise: profile a real script you run often, find its top function by tottime, and see whether the result matches what you would have guessed.