Iterators change the question

Learning outcomes

After this tutorial

Distinguish iterables, iterators, and generator objects
Predict how iter, next, and for consume a source
Build lazy pipelines with generators and itertools
Recognize one-shot, memory, and debugging tradeoffs

Suppose a log file contains ten million lines and you need the first three errors. A collection-shaped question asks, "How do I load the lines, filter all errors, and take three?" An iterator-shaped question asks, "What is the next error, and have I seen enough?"

from itertools import islice


def error_lines(lines):
    for line in lines:
        if "ERROR" in line:
            yield line.rstrip("\n")


sample = iter([
    "INFO starting\n",
    "ERROR disk full\n",
    "INFO retrying\n",
    "ERROR still full\n",
    "ERROR giving up\n",
    "ERROR never requested\n",
])

print("[result] first three errors:", list(islice(error_lines(sample), 3)))
print("[state] next unconsumed line:", next(sample).rstrip("\n"))

['ERROR disk full', 'ERROR still full', 'ERROR giving up']
ERROR never requested

The pipeline stopped as soon as it had an answer. It did not build a list of every line or examine the final line. This is the practical shift behind Python's iteration protocol: code can request values over time instead of requiring an entire collection up front.

Version note. The examples target Python 3.10 through 3.14 and were verified on CPython 3.14. The iterator protocol is a Python language feature. Details about generator frames and object sizes can vary between implementations and releases.

Two protocols, three useful words

An iterable is an object from which Python can obtain an iterator. Lists, tuples, strings, dictionaries, sets, files, ranges, and many user-defined objects are iterable. Informally, an iterable answers: "Can iteration start here?"

An iterator is the stateful object that produces one value at a time. It implements __next__(), which returns the next value or raises StopIteration, and __iter__(), which returns the iterator itself. An iterator answers: "What comes next from my current position?"

A container usually stores values and can often create fresh iterators, but storage is not required for iterability. range(1_000_000) is iterable without storing one million integer objects. A generator can compute values on demand without representing a reusable collection at all.

The distinction is visible with iter():

numbers = [10, 20, 30]

first = iter(numbers)
second = iter(numbers)

print(first is second)
print(next(first), next(first))
print(next(second))
print(iter(first) is first)

False
10 20
10
True

The list is an iterable. Each call to iter(numbers) creates an independent list iterator. first is already an iterator, so iter(first) returns first, preserving its current position.

This gives a practical diagnostic:

def classify(value):
    candidate = iter(value)
    return "iterator" if candidate is value else "re-iterable"


print(classify([1, 2, 3]))
print(classify(iter([1, 2, 3])))

It is a useful demonstration, not a complete taxonomy. Some unusual user-defined types can choose different behavior. When an API contract matters, use collections.abc.Iterable and Iterator for runtime checks or type annotations, and document whether repeated traversal is supported.

What `for` is doing

At the protocol level, this loop:

for item in ["red", "green"]:
    print("[step] for-loop item:", item)

behaves roughly like this:

iterator = iter(["red", "green"])

while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    print("[step] manual-loop item:", item)

Real bytecode is not required to be this source transformation, but the model is accurate: iter() obtains an iterator, next() repeatedly requests values, and StopIteration means normal exhaustion. for catches that exception for you.

The two-argument form of next() supplies a sentinel instead of exposing exhaustion:

values = iter([4])

print("[result] first next call:", next(values, "missing"))
print("[result] next call after exhaustion:", next(values, "missing"))

4
missing

Use a private sentinel when None or another ordinary value could be valid:

missing = object()
value = next(iter([None]), missing)

if value is missing:
    print("empty")
else:
    print(f"found {value!r}")

Do not generally call __next__() directly. next(value) states the protocol operation and works with the optional default.

`iter()` can turn repeated calls into iteration

The less familiar two-argument form, iter(callable, sentinel), repeatedly calls a zero-argument callable. It yields each result until one compares equal to the sentinel, then stops without yielding that sentinel. This is useful when an API reports completion by returning a special value instead of raising StopIteration.

from functools import partial
from io import BytesIO


stream = BytesIO(b"abcdefghij")
read_four = partial(stream.read, 4)

for block in iter(read_four, b""):
    print("[step] bytes block:", block)

b'abcd'
b'efgh'
b'ij'

BytesIO.read(4) returns up to four bytes and eventually returns b"" at end of input. The callable form of iter() adapts that convention into the same protocol a for loop expects. It keeps block processing bounded without a manual while True and break.

Choose the sentinel carefully. The stopping test uses equality, so an ordinary data value equal to the sentinel ends iteration. Exceptions from the callable still propagate; the sentinel form does not turn failures into normal exhaustion. Also ensure the callable can eventually return the sentinel. A callable that returns valid values forever creates an infinite iterator, which is useful only when some downstream operation imposes a limit.

This overload and the ordinary iter(iterable) overload solve opposite adaptation problems. One asks an object for its iterator. The other wraps a repeated-call convention as an iterator. Both let downstream code depend on next value or exhaustion rather than the source's concrete shape.

A small iterator class

Implementing the protocol makes its statefulness concrete:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


countdown = Countdown(3)
print("[result] first countdown value:", next(countdown))
print("[result] remaining countdown values:", list(countdown))
print("[result] exhausted countdown values:", list(countdown))

3
[2, 1]
[]

Countdown combines the iterable and iterator roles in one object. That is appropriate for a single traversal, but it creates one-shot behavior. A reusable countdown would instead store only start and have __iter__() return a fresh iterator, perhaps iter(range(self.start, 0, -1)).

One-shot consumption is part of the contract

An exhausted iterator stays exhausted. Consumers such as list(), tuple(), sum(), min(), membership tests, unpacking, and loops advance it. Even a test that appears observational can consume values:

values = iter([1, 3, 5, 8, 9])

print("[check] iterator contains 8:", 8 in values)
print("[state] values remaining after membership test:", list(values))

True
[9]

The membership operation advanced through 1, 3, 5, and 8. It stopped after finding a match, leaving only 9.

Common failure modes follow from forgetting this state:

Logging with list(iterator) drains the values before production code sees them.
Computing sum(values) and then len(list(values)) measures an empty remainder.
Passing the same iterator to two consumers makes their results depend on call order.
Retrying a failed operation with the same iterator resumes partway through rather than starting over.
Asking whether a value occurs in an infinite iterator may never return if the value never appears.

If several independent passes are required, accept an iterable that can create fresh iterators, recreate the source, or materialize it deliberately. itertools.tee() can split one input into independent-looking iterators, but it buffers values when one copy moves ahead of another. A large gap can consume as much memory as materializing the lagging portion, and tee iterators are not threadsafe.

Generator functions keep suspended execution

The yield keyword turns a function into a generator function. Calling it does not run its body immediately; it returns a generator object. Each next() resumes execution until the next yield, then suspends it while preserving local variables and the instruction position.

def running_totals(values):
    total = 0
    for value in values:
        total += value
        yield total


totals = running_totals([5, 7, 2])
print("[result] first running total:", next(totals))
print("[result] second running total:", next(totals))
print("[result] remaining running totals:", list(totals))

5
12
[14]

Conceptually, the generator object owns a suspended execution frame containing total, value, the source iterator, and where execution should continue. On CPython you can inspect totals.gi_frame, but that attribute and the frame's representation are implementation-oriented debugging details, not a design surface to build application logic around.

A normal return ends a generator. Python translates that completion into StopIteration for the consumer. PEP 479 also prevents an accidental StopIteration raised inside generator code from silently looking like ordinary completion: it becomes RuntimeError. Usually the right way to stop a generator is return, not raise StopIteration.

Generators also have send(), throw(), and close(), and yield from can delegate to another iterable. Those tools matter for coroutine-like protocols and cleanup, but ordinary data pipelines should start with iteration and yield; introducing two-way communication makes control flow harder to follow.

Laziness moves work and memory

A list comprehension completes all work and stores every result before the next statement runs. A generator expression performs work as values are requested:

def square(value):
    print(f"[event] squaring value: {value}")
    return value * value


eager = [square(n) for n in range(3)]
print("[state] eager collection built")

lazy = (square(n) for n in range(3))
print("[state] lazy generator built without consuming values")
print("[result] first lazy square:", next(lazy))

The eager version prints three messages before eager built. The lazy version prints nothing until next(lazy), then computes only the first square.

This can reduce peak memory from proportional to the number of results to proportional to pipeline state. It can also permit infinite sources and early stopping. It does not mean generators are automatically faster. Each value still requires Python-level resumption and dispatch, while a materialized list can be faster to traverse repeatedly. A generator object and its suspended state also consume some memory; laziness is an allocation strategy, not free storage.

Choose laziness when input is large or unbounded, results are consumed once, work is expensive, or consumers often stop early. Choose materialization when data is modest, repeated traversal or random access is needed, a stable snapshot matters, or separating production from consumption makes failures easier to diagnose.

Timing changes too. File reads, parsing errors, database requests, and side effects happen during consumption, possibly far from where the generator was created. Keep resources open for the whole consumption period, and avoid hiding important side effects inside a pipeline merely because yield makes it concise.

Build pipelines with `itertools`

The standard library's itertools module provides small iterator building blocks implemented for composability. A realistic pipeline can remain bounded:

from itertools import chain, islice


cached = ["ok: cache", "error: stale"]
live = iter(["ok: live", "error: timeout", "error: refused"])

events = chain(cached, live)
errors = (event for event in events if event.startswith("error:"))
first_two = list(islice(errors, 2))

print("[result] first two errors:", first_two)
print("[state] next unconsumed live event:", next(live))

['error: stale', 'error: timeout']
error: refused

chain traverses sources in order. The generator expression filters. islice requests at most two matches. Only the final list materializes the answer.

Several tools recur in production code:

islice(iterable, n) takes a bounded prefix without slicing storage.
chain(*iterables) presents several sources as one sequence of values.
count(), cycle(), and repeat() create infinite iterators; always pair them with a stopping condition.
takewhile() and dropwhile() split behavior around the first predicate failure, not every failure.
batched(iterable, n) groups input into tuples of up to n items.
pairwise() yields overlapping adjacent pairs.
groupby() groups consecutive equal keys, not all equal keys across unsorted input.

That last distinction causes frequent bugs:

from itertools import groupby


records = ["red", "blue", "red"]
groups = [(key, list(group)) for key, group in groupby(records)]
print("[result] consecutive groups:", groups)

The output contains three groups because the two "red" runs are separated. Sort first only if global grouping and reordered input are actually acceptable; otherwise use a dictionary-based accumulation.

Boundaries make lazy code reliable

Iterator pipelines are easiest to reason about when ownership is clear. The function that creates a one-shot iterator should document that fact. The layer that needs a concrete snapshot should call list() explicitly. Avoid APIs that sometimes return a list and sometimes a generator based on input size; callers cannot safely infer consumption behavior.

When debugging, sample without accidentally destroying the only source. If recreation is cheap, create a fresh iterator for diagnostics. If not, intentionally buffer a small prefix and put it back with chain:

from itertools import chain, islice


source = iter(range(10))
preview = list(islice(source, 3))
source = chain(preview, source)

print("[result] buffered preview:", preview)
print("[result] reconstructed full source:", list(source))

For file and network iterators, define resource ownership as carefully as value ownership. Returning a generator that reads from a file opened by a completed with block returns a broken pipeline. Either let the generator own the with block for the duration of iteration or let the caller own the already-open resource.

Exercises: test the protocol

Predict every line printed by the eager-versus-lazy square() example before running it. Explain which statement triggers each call.
Write a reusable Countdown iterable whose __iter__() returns a fresh iterator. Verify that two loops both produce the full countdown.
Create one iterator over range(10). Call next(), sum(), and list() in sequence, and predict each result.
Use itertools.islice and a generator expression to find the first five multiples of 17 from itertools.count(1) without an unbounded loop or list.
Demonstrate groupby() on unsorted records, then decide whether sorting or dictionary accumulation better matches global grouping for your example.
Return lines lazily from a file while ensuring the file remains open during iteration and closes when iteration finishes.

Keep this model

An iterable can produce an iterator. An iterator is a stateful, one-way cursor whose __next__() either yields the next value or raises StopIteration. A generator is an iterator implemented by suspending a function's execution frame at each yield.

Once an API depends on that protocol rather than a list, it can stream, stop early, combine sources, and represent unbounded data. The cost is equally concrete: consumption mutates position, effects happen later, repeated traversal is unavailable unless designed in, and buffering can reappear in tools such as tee or list.

The question is not "Should this code use generators?" Ask instead: "Does this consumer need all values, all at once, more than once?" The answer tells you whether laziness clarifies the system or merely moves its complexity.