beginnerintermediateadvanced

Python · Section 6

Iterators and Generators

The protocol behind every for loop, and yield — the shortcut that implements it for you

iter() once, next() repeatedlyexactly what a for loop does
yield pauses, does not returnstate survives between calls
Lazy: one item at a timeflat memory, however long the source
close() raises GeneratorExita finally block still runs

1.Iterable vs iterator

An iterable is anything iter() can be called on — a list, a string, a generator function's result. An iterator is what iter() returns: an object that remembers a position and produces one value per next() call.

Key features

Iterable: defines __iter__, returning an iterator — a list, dict, str, set, range
Iterator: defines __next__, returning one value per call, usually __iter__ too
iter(an_iterable) produces a fresh iterator each time, never the same object twice
Every iterator is itself iterable, but not every iterable is an iterator
A for loop only requires an iterable — it calls iter() on it once, automatically

Telling the two apart on a real object

Telling the two apart on a real object
Check	A list	A list's iterator
hasattr(x, "__iter__")	True	True
hasattr(x, "__next__")	False	True
iter(x) is x	False — a new iterator each time	True — an iterator is its own iterator

Command

python

iter(obj)        # works if obj has __iter__ -> obj is an iterable
next(obj)        # works if obj has __next__ -> obj is an iterator

Iterable has __iter__ and can be asked for an iterator. Iterator has __next__ and produces values one at a time.

2.Iterator protocol

The iterator protocol is the two-method contract — __iter__ and __next__ — that every for loop, comprehension, and unpacking relies on. Any object honouring it works everywhere Python expects something iterable, without special-casing.

Key features

Two parts: __iter__ returns an iterator; __next__ returns a value or raises StopIteration
Anything satisfying it works with for, list(), sum(), unpacking — no special-casing
A generator function automatically produces an object satisfying the full protocol
Hand-writing a class with __iter__/__next__ is the manual route to the same contract
Duck typing, not inheritance — no base class marks an object as "an iterator"

Two ways to satisfy the same protocol

Two ways to satisfy the same protocol
Approach	How __iter__/__next__ get implemented
A class with dunders	Written by hand — __iter__ returns self or a helper object; __next__ advances state and raises StopIteration
A generator function	Implemented automatically the moment the function body contains yield

Command

python

class Name:
    def __iter__(self):
        ...
        return an_iterator

    def __next__(self):
        ...
        return value  # or: raise StopIteration

The protocol is __iter__ + __next__ — satisfied by hand-written dunders, or automatically by any function containing yield.

3.Generator functions and yield

A function with yield anywhere inside it is a generator function. Calling it does not run the body — it returns a generator object that runs from the top up to the next yield each time you ask it for a value.

Key features

A single yield anywhere in a function body makes it a generator function
Calling a generator function returns a generator object — no code runs yet
next(gen) runs the body until the next yield, returns that value, pauses there
Local variables and the execution point are preserved across pauses
Reaching the end of the body raises StopIteration — the generator is exhausted

Generator function vs. a normal function that builds a list

Generator function vs. a normal function that builds a list
Question	Normal function returning a list	Generator function with yield
When does the body run?	Entirely, on the call	In bursts, one per next() call
What does calling it return?	The finished list	A generator object, unstarted
Memory for N items	O(N) — every item exists at once	O(1) — one item exists at a time
Can you consume it twice?	Yes — the list still exists	No — exhausted after one full pass

Calling a generator function vs. calling next()

Call the function

body does NOT run yet

Returns a generator

paused before the first line

next() runs to yield

pauses there, returns the value

Body ends

raises StopIteration

Command

python

def name(args):
    ...
    yield value          # pauses here, hands back value
    ...                  # resumes here on the next next() call
    # falling off the end (or a bare "return") raises StopIteration

A function with yield returns a generator when called. Its body runs in bursts, one per next(), pausing at each yield.

4.yield from

yield from sub_iterable yields every item of sub_iterable, in order, as though the outer generator looped and yielded each one itself. It also forwards send()/throw()/close() to the sub-generator.

Key features

yield from x is equivalent to for item in x: yield item — for plain iteration
Works on any iterable: a list, a string, a range, or another generator
The natural way to delegate recursively — a generator calling itself through yield from
Also forwards .send(), .throw() and .close() through to the sub-generator
The delegating generator resumes its own code only after the sub-iterable is exhausted

Command

python

def outer():
    yield "before"
    yield from inner_iterable   # yields every item of inner_iterable here
    yield "after"

yield from sub delegates the entire iteration to sub, and send()/throw()/close() pass through to it too.

5.Lazy evaluation

Lazy evaluation means a value is computed only when something asks for it, not when the expression is written. A generator expression is lazy; a list comprehension is eager.

Key features

A list comprehension runs expr for every x immediately, building the full list
A generator expression runs expr for one x only when next() or a for loop asks
A side effect inside expr — print(), a network call — reveals the difference directly
Lazy means flat memory: one item exists at a time, however long data is
Prefer [] when reusing the result or needing len(); prefer () for a single pass

Eager vs lazy, side by side

Eager vs lazy, side by side
Question	result = [transform(x) for x in data]	result = (transform(x) for x in data)
When does transform() run?	Immediately, for every x, on this line	Only when result is iterated, one x at a time
What is result right after?	A full list of results	A generator object — nothing computed yet
Can it be consumed twice?	Yes	No — one-shot, exhausted after a full pass
Memory for a million items	O(n) — every result stored	O(1) — one result in flight

When transform() actually runs

List comprehension

runs transform() for every item now

Generator expression

stores the loop, runs nothing yet

Iterated later

transform() runs one item at a time

Command

python

eager = [transform(x) for x in data]   # every item computed on this line
lazy = (transform(x) for x in data)      # nothing computed until iterated

Eager (list comprehension) computes everything now. Lazy (generator expression) computes nothing until iterated, one item at a time.

6.Generator pipelines

A generator pipeline chains several generator functions, each wrapping the one before it. An item flows through every stage before the next item enters the pipeline at all — no stage waits for the whole source to finish first.

Key features

Each stage is a generator function that takes an iterable and yields transformed items
Stages compose by nesting calls: stage3(stage2(stage1(source)))
No stage runs to completion before the next starts — items flow through one at a time
A failure in the source surfaces only when the bad item is reached, not at construction
Reads as a sequence of named steps, not one large loop with everything inlined

Command

python

def stage_one(source):
    for item in source:
        yield transform(item)

def stage_two(items):
    for item in items:
        if condition(item):
            yield item

pipeline = stage_two(stage_one(raw_source))   # nothing runs until iterated

Chain generator functions by nesting calls. An item moves through every stage before the next item enters the pipeline.

7.Memory-efficient processing

Processing data with generators keeps memory flat: one item exists at a time, no matter how many items the source has. Building intermediate lists instead means every item exists at once, which does not scale to a large or unbounded source.

Key features

A generator holds a loop and a position — flat memory regardless of length
A list from the same source holds every item at once — memory grows with length
Keeping every pipeline stage as a generator preserves flat memory end to end
The cost of lazy, one-pass processing: no going back, no len() in advance
A source too large for memory, or genuinely unbounded, needs a generator

Command

python

# grows with the source
rows = [transform(line) for line in open(path)]

# flat, regardless of source size
rows = (transform(line) for line in open(path))
for row in rows:
    handle(row)

A pipeline built entirely of generators keeps memory flat. Converting any stage to a list brings the whole source into memory at that point.

8.Sending values into generators

gen.send(value) resumes a paused generator and delivers value as the result of the yield expression it paused on. next(gen) is exactly gen.send(None).

Key features

received = yield value pauses, hands out value, and resumes with what send() delivers
next(gen) is equivalent to gen.send(None) — plain iteration sends nothing in
A generator needs priming with next() before the first real send() — no yield waits yet
send(x) on a not-yet-started generator raises TypeError
Underlies coroutine-style generators; async def/await replaced it for real async code

Command

python

def name():
    received = yield initial_value   # pauses here, returns initial_value
    ...                                # resumes here, received == whatever send() delivered

gen = name()
first = next(gen)          # prime it - runs to the first yield
result = gen.send(value)    # delivers value, resumes, runs to the next yield

send(value) resumes a generator, making value the result of the yield it paused on. Prime with next() first.

9.Generator cleanup

gen.close() raises GeneratorExit at the generator's current paused point, which a try/finally block can catch to release a resource. After close(), the generator is permanently exhausted — a later next() raises StopIteration.

Key features

gen.close() raises GeneratorExit inside the generator, at its current yield point
A try/finally around the loop runs on close(), same as for any other exception
After close(), the generator is exhausted — next(gen) raises StopIteration
A generator that yields again after catching GeneratorExit raises RuntimeError
Garbage collection calls close() eventually, but do not rely on its timing

Command

python

def managed():
    try:
        while True:
            yield "data"
    finally:
        release_resource()   # runs on close(), on completion, or on garbage collection

gen = managed()
next(gen)
gen.close()   # raises GeneratorExit inside managed(), runs the finally block

close() raises GeneratorExit at the generator's paused point — a finally block runs, then it is exhausted for good.

Spotted a mistake on this page? Tell us.

Read Sheet