Python makes it dangerously easy to produce a precise answer to the wrong question. Put two expressions into timeit, copy the smaller number into a pull request, and an implementation detail has become a performance claim.

A useful benchmark is not primarily a stopwatch. It is a controlled experiment with a stated question, representative work, observable outcomes, and enough environmental detail for someone else to challenge it. This tutorial builds that discipline from the standard library upward.

Test environment. All recorded results below came from the repository's .venv/bin/python: CPython 3.14.7, arm64, macOS 26.5.2, traditional GIL enabled, optimization level 0, non-debug build, compiled with Clang 21.0.0. The machine was on mains power with ordinary background applications present. Your numbers should differ. The commands and conclusions are the reproducible part.

Experiment 1: write down what ran

Before timing code, capture enough context to explain a disagreement:

python -VV
python - <<'PY'
import platform
import sys

print(platform.platform())
print(platform.machine())
print("implementation:", platform.python_implementation())
gil_enabled = getattr(sys, "_is_gil_enabled", lambda: "not exposed")()
print("GIL enabled:", gil_enabled)
print("optimization level:", sys.flags.optimize)
print("debug build:", hasattr(sys, "gettotalrefcount"))
PY

Our run reported CPython 3.14.7 on macOS-26.5.2-arm64-arm-64bit-Mach-O, with the GIL enabled, optimization level zero, and no debug-build-only gettotalrefcount.

Record the Python executable, exact version, implementation, architecture, operating system, dependency versions, input description, and benchmark command. For serious comparisons, also record CPU model, power mode, container limits, and whether the machine was otherwise idle. A debug build, a free-threaded build, PyPy, and ordinary CPython are not interchangeable test subjects.

Python guarantee versus environment. Python defines program behavior, not how many nanoseconds an operation takes. Even complexity claims usually describe growth, not constants. Timing is an observation about a particular implementation, version, build, machine, operating state, and workload.

Start every benchmark with a sentence that can be falsified: "For 10,000 already-decoded records, implementation B reduces median request CPU time without changing output" is a question. "B is faster" is not.

Experiment 2: inspect the clock

Elapsed time should come from a monotonic performance clock, not calendar time:

import time

info = time.get_clock_info("perf_counter")
print(info.implementation)
print("monotonic:", info.monotonic)
print("adjustable:", info.adjustable)
print("reported resolution:", info.resolution)

start = time.perf_counter_ns()
time.sleep(0.01)
print("elapsed ns:", time.perf_counter_ns() - start)

Our macOS run used mach_absolute_time(), was monotonic and non-adjustable, and reported a resolution of about 41.7 ns. The sleep took slightly more than 10 million ns.

time.perf_counter() measures elapsed wall time and includes sleep. time.process_time() measures user and system CPU consumed by the process and excludes sleep. Neither is universally better. Wall time answers latency questions, including waiting and scheduling. Process time can help isolate CPU consumption, but says little about user-visible latency or work performed elsewhere.

Use the _ns variants when integer nanoseconds make arithmetic or storage clearer. The suffix does not make the underlying clock more accurate. Reported resolution is also not a promise that one tiny operation can be timed accurately; reading the clock and scheduling the process both have costs.

Experiment 3: let timeit build the inner loop

For a small synchronous operation, begin with the command-line interface:

python -m timeit -r 7 -s 'values = range(100)' 'sum(values)'
python -m timeit -r 7 -s 'values = range(100)' 'sum(x for x in values)'

timeit compiles a timing function, runs setup outside the measured loop, automatically chooses a loop count on the CLI, and repeats the measurement. It uses time.perf_counter() by default. On our machine, direct sum(values) was hundreds of nanoseconds per loop; the generator version took several microseconds. Those numbers apply to this exact input and interpreter, not to every reduction.

Prefer a statement string when measuring extremely small code because passing a callable adds another Python call. You can measure that difference directly:

import timeit

values = range(100)

def run():
    return sum(values)

string_runs = timeit.repeat(
    "sum(values)", globals=globals(), number=500_000, repeat=5
)
callable_runs = timeit.repeat(run, number=500_000, repeat=5)

print(min(string_runs) / 500_000 * 1e9)
print(min(callable_runs) / 500_000 * 1e9)

We observed about 307 ns for the statement and 319 ns for the callable. For a 20 ms operation, that difference is irrelevant, and a callable may produce a much clearer benchmark. Harness overhead matters only relative to the work.

Experiment 4: prove what setup excludes

Setup is executed before each timing run but excluded from the duration returned by Timer.timeit():

import time
import timeit

timer = timeit.Timer(
    "x += 1",
    setup="import time; time.sleep(0.05); x = 0",
)

wall_start = time.perf_counter()
measured = timer.timeit(number=1_000_000)
wall = time.perf_counter() - wall_start

print(f"reported: {measured:.4f}s")
print(f"wall:     {wall:.4f}s")

Our run reported 0.0099s, while the surrounding wall measurement was 0.0648s. That is correct behavior, but it can support a false claim if setup contains work production must perform.

Move imports, fixture construction, and one-time compilation into setup only when the real system can amortize them similarly. If every request parses a schema, excluding parsing does not benchmark request latency. Conversely, repeatedly importing a module when a long-lived process imports it once overstates steady-state cost.

State whether you are measuring cold start, first call, warmed steady state, throughput over a batch, or end-to-end latency. These are different products, not alternative ways to label the same number.

Experiment 5: preserve the whole result vector

Use autorange() when driving Timer yourself, then inspect every repeat:

import timeit

timer = timeit.Timer("sum(values)", setup="values = range(100)")
number, elapsed = timer.autorange()
runs = timer.repeat(repeat=7, number=number)

print("loops per repeat:", number)
print([round(run / number * 1e9, 1) for run in runs])

Our output was:

loops per repeat: 1000000
[330.3, 348.4, 327.2, 318.7, 315.6, 317.6, 314.8]

The timeit documentation recommends the minimum as a useful lower bound because slower repeats often reflect interference. That is not permission to publish only the luckiest run. Keep the vector, describe the summary, and investigate wide or multimodal results. For application benchmarks with process-to-process variation, medians, percentiles, confidence intervals, and many fresh processes may answer better questions than timeit's minimum.

Do not manufacture precision. Reporting 314.812739 ns suggests knowledge the experiment does not contain. A relative change smaller than normal run-to-run spread is evidence to gather more data, not a victory to round creatively.

Randomize or alternate A/B order when thermal state, caches, or background load can favor whichever candidate always runs first. Run each candidate in fresh processes if global interpreter state, imports, allocators, or specialization can leak between cases.

Experiment 6: include garbage collection when it belongs

timeit temporarily disables cyclic garbage collection during a timing run. Reference counting still happens in CPython, but cyclic collection pauses do not unless setup re-enables GC.

import timeit

statement = "x = []; x.append(x)"

without_gc = timeit.repeat(
    statement, number=20_000, repeat=5
)
with_gc = timeit.repeat(
    statement, setup="gc.enable()", number=20_000, repeat=5
)

to_ns = lambda runs: [round(t / 20_000 * 1e9, 1) for t in runs]
print("GC disabled:", to_ns(without_gc))
print("GC enabled: ", to_ns(with_gc))

Our per-loop vectors were approximately [32.4, 23.3, 25.5, 24.7, 24.7] ns and [131.7, 75.3, 70.3, 99.4, 83.6] ns. The statement deliberately creates unreachable cycles, so excluding collection removes real eventual work and lets garbage accumulate until the harness restores GC.

For code that creates no cycles, disabling GC can improve comparability. For allocation-heavy request handling, latency-sensitive services, or an operation that creates cycles, measure with production-like collection settings too. This default is documented timeit behavior, not a language guarantee and not proof that "Python has no GC cost."

Experiment 7: catch the mutating-input lie

A benchmark loop may quietly transform its own input:

import random
import timeit

base = list(range(10_000))
random.Random(42).shuffle(base)
timer = timeit.Timer(
    "data.sort()",
    setup="data = base.copy()",
    globals=globals(),
)

first = timer.repeat(repeat=5, number=1)
amortized = timer.repeat(repeat=5, number=100)
fresh = timeit.repeat(
    "sorted(base)", globals=globals(), repeat=5, number=100
)

us = lambda runs, n: [round(t / n * 1e6, 1) for t in runs]
print("first sort:", us(first, 1))
print("same list 100 times:", us(amortized, 100))
print("fresh result each time:", us(fresh, 100))

Our best results were about 683 us for the first in-place sort, 24 us per call when averaging 100 calls on the same list, and 677 us when producing a fresh sorted result each time. The 24 us result mostly measures sorting an already sorted list. It is precise and useless for random input.

Audit every input for mutation, cache population, lazy initialization, file position, database state, memoization, and consumed iterators. Setup executes once per repeat, not once per inner-loop iteration. Sometimes the fix is number=1 with many fresh-process repeats. Sometimes input reset belongs inside the measured operation because production pays for it. Do not subtract fixture cost unless subtraction corresponds to a real boundary.

Also verify outputs. A candidate that returns less data, skips validation, or raises and catches an error may win by doing a different job.

Experiment 8: batch work larger than timer overhead

Timing one tiny call between two clock reads mostly measures clock reads and noise:

import time

def tiny(x):
    return x + 1

single = []
for _ in range(20):
    start = time.perf_counter_ns()
    tiny(1)
    single.append(time.perf_counter_ns() - start)

start = time.perf_counter_ns()
for _ in range(100_000):
    tiny(1)
batched = (time.perf_counter_ns() - start) / 100_000

print(single)
print(f"batched average: {batched:.1f} ns")

Our single-call samples ranged from 83 ns to 1542 ns, while the batch averaged 56.2 ns per iteration. The latter still includes loop overhead, and subtracting an empty loop can amplify error. Usually the right move is not elaborate subtraction but making the measured unit large enough that fixed harness cost is negligible.

Be alert to compiler and interpreter behavior. CPython 3.14 specializes hot bytecode as it runs, so a large timeit loop usually describes warmed code. Other Python implementations may have much longer JIT warmup. If first-call latency matters, use fresh processes. If steady state matters, warm both candidates consistently and say so.

Profile to locate work, benchmark to compare it

cProfile answers where a Python program spent instrumented time and how often functions were called. It adds overhead unevenly, especially across Python and C code, so its own documentation explicitly says it is not a benchmarking tool.

Use it to find candidates:

python -m cProfile -o profile.data your_program.py
python -m pstats profile.data

Then benchmark a realistic boundary without the profiler attached. For I/O-heavy systems, add load tests that preserve concurrency, payloads, network behavior, and service dependencies. A nanosecond microbenchmark cannot predict queueing, tail latency, memory pressure, or database contention.

A practical decision protocol

Before accepting an optimization, require this short record:

  1. Question: workload, metric, and acceptable tradeoff.
  2. Equivalence: tests or assertions proving both candidates produce required outcomes.
  3. Inputs: sizes, distributions, hit rates, and mutation/reset policy.
  4. Boundary: exactly what setup and teardown are excluded, with production justification.
  5. Environment: exact executable, versions, build mode, hardware, and relevant settings.
  6. Method: clock, warmup, loops, repeats, process isolation, and order.
  7. Results: raw observations plus an explicitly named summary and units.
  8. Decision: effect size, uncertainty, maintenance cost, and whether the result matters end to end.

A 10% microbenchmark improvement in code consuming 1% of runtime can improve the application by at most about 0.1% before secondary effects. A simpler algorithm, fewer network calls, or better data layout usually deserves attention before shaving an opcode from cold code.

Exercises

  1. Benchmark membership in a list and set at sizes 4, 100, and 10,000. Include present and absent values, fix PYTHONHASHSEED, and explain which conclusions are semantic versus CPython-specific.
  2. Write a benchmark whose input is an iterator. First accidentally consume it across loops, then repair the experiment and compare the results.
  3. Measure a function with perf_counter() and process_time() while it sleeps. State which result answers latency and which answers local CPU consumption.
  4. Re-run the cyclic-allocation experiment with production GC thresholds. Report the full vectors instead of only minima.
  5. Choose one hot function from cProfile, construct a representative benchmark around it, and estimate the maximum possible end-to-end gain before changing code.

Keep this model

A benchmark is a claim with an audit trail. Choose the metric from the user-visible question, preserve realistic work, control state, batch below-clock-resolution operations, repeat measurements, and expose variability. Separate Python semantics from CPython timing and exact-version implementation behavior.

The most dangerous benchmark is not noisy. It is stable, repeatable, and measuring the wrong thing.

Primary sources