Filter concepts by levelShowing all levels.

Python · Section 6

Iterators and Generators

Level
intermediate
Read
58 min
Concepts
9

The iterator protocol is the two-method contract — __iter__ and __next__ — behind every for loop, comprehension, and unpacking. Generator functions satisfy that protocol automatically: a function containing yield can pause mid-execution and resume exactly where it left off, computing each item only when asked. This section covers the protocol as a pattern, writing and chaining generator functions, the lazy-versus-eager tradeoff that decides when to reach for one, and the two-way channel (send, close) a generator supports beyond plain iteration.

This section
Advanced Python Series - Iterators Vs GeneratorsKrish Naik

What is true here

  1. An iterable produces iterators (__iter__); an iterator produces values (__next__) — related but different roles.
  2. A function containing yield anywhere becomes a generator function — calling it never runs the body, only next() does, in bursts.
  3. A generator expression is lazy — one item computed per request; a list comprehension is eager — every item computed immediately.
  4. yield from delegates to a sub-iterable and forwards send()/throw()/close() through to it, not just plain values.
  5. gen.close() raises GeneratorExit at the generator's paused point, so a finally block can release a resource before it becomes unusable.

What you will be able to do

  • Tell an iterable from an iterator, and explain why a for loop only requires the former
  • Write a generator function and predict exactly when each line of its body runs
  • Choose a list comprehension or a generator expression for a given job, based on eagerness and memory
  • Chain generator functions into a pipeline that keeps memory flat across every stage
  • Use yield from to delegate to a sub-iterable, and gen.send()/gen.close() to drive a generator beyond plain iteration
From the protocol to a lazy pipeline
yieldsatisfies itcomputeson demandstagescompose

Iterator protocol

__iter__ + __next__ — the contract

Generator function

yield implements the protocol automatically

Lazy, one item at a time

flat memory, however long the source

Chained into a pipeline

each item flows through every stage before the next starts

  • Iterator protocol — __iter__ + __next__ — the contract
    • leads to Generator function (yield satisfies it)
  • Generator function — yield implements the protocol automatically
    • leads to Lazy, one item at a time (computes on demand)
  • Lazy, one item at a time — flat memory, however long the source
    • leads to Chained into a pipeline (stages compose)
  • Chained into a pipeline — each item flows through every stage before the next starts

The iterator protocol

What separates something that can be iterated from something that does the iterating, and the contract a for loop actually relies on.

Iterable vs iterator

standardintermediate

An iterable is anything iter() can be called on — a list, a string, a generator function's result. An iterator is what iter() returns: an object that remembers a position and produces one value per next() call.

Think of it as

A book is iterable — you can open it and start reading. A bookmark is an iterator — it tracks one specific reading position inside one specific reading session. The same book supports many independent bookmarks at once; each iterator has its own position, even over the same iterable.

python
iter(obj)        # works if obj has __iter__ -> obj is an iterable
next(obj)        # works if obj has __next__ -> obj is an iterator

What we're doing: Write a class that is iterable without being an iterator, and show the same instance supports two independent, simultaneous loops.

fibonacci.pypython
class Fibonacci:
    def __init__(self, count):
        self.count = count

    def __iter__(self):
        a, b = 0, 1
        yielded = 0
        while yielded < self.count:
            yield a
            a, b = b, a + b
            yielded += 1


fib = Fibonacci(5)
print(list(fib))
print(list(fib))  # works again - __iter__ builds a fresh generator each call
for n in fib:
    print(n, end=' ')
5
__iter__ is a generator function (it contains yield), so calling it does not run the body — it returns a fresh generator object, which is the iterator.
12
Fibonacci defines __iter__ but no __next__, so an instance is an iterable, not an iterator — you cannot call next(fib) directly.
14
A second list(fib) works because __iter__ runs again, building a brand-new generator with its own a, b, yielded — nothing carries over from the first call.
Output
[0, 1, 1, 2, 3]
[0, 1, 1, 2, 3]
0 1 1 2 3

Why this works: Because __iter__ is called once per iteration attempt and each call builds an independent generator, the SAME Fibonacci instance supports being iterated as many times as needed, from scratch every time. That independence is exactly what separates an iterable from an iterator: the iterable is reusable, the iterator it hands out is not.

Treating an iterable as though it were already an iterator

Wrong

python
class BadCounter:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __next__(self):          # __next__ with no __iter__
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current

for n in BadCounter(3):
    print(n)

Better

python
class GoodCounter:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current

for n in GoodCounter(3):
    print(n)

What you see: TypeError: 'BadCounter' object is not iterable

Why: A for loop calls iter() on its argument before it ever calls next() — that is a hard requirement, not an optimization. BadCounter defines __next__ but not __iter__, so iter(BadCounter(3)) has nothing to call and Python raises immediately, before a single value is produced. Adding __iter__ that returns self is what makes the object both iterable and its own iterator.

Telling the two apart on a real object

Telling the two apart on a real object
CheckA listA list's iterator
hasattr(x, "__iter__")TrueTrue
hasattr(x, "__next__")FalseTrue
iter(x) is xFalse — a new iterator each timeTrue — an iterator is its own iterator

Together

python
nums = [1, 2, 3]
print(hasattr(nums, '__iter__'), hasattr(nums, '__next__'))  # True False - iterable, not an iterator

it = iter(nums)
print(hasattr(it, '__iter__'), hasattr(it, '__next__'))      # True True - an iterator
print(iter(it) is it)                                         # True - iter() on an iterator returns itself

Remember: Iterable has __iter__ and can be asked for an iterator. Iterator has __next__ and produces values one at a time.

See also: iterator protocol · iter next · iterator protocol dunders

Iterator protocol

standardintermediate

The iterator protocol is the two-method contract — __iter__ and __next__ — that every for loop, comprehension, and unpacking relies on. Any object honouring it works everywhere Python expects something iterable, without special-casing.

Think of it as

A protocol is a shape, not a family tree. A list, a file, a database cursor, and a generator share no common ancestor class, but every one of them satisfies the iterator protocol — so a for loop treats them identically. Satisfying the shape is what matters, not what the object otherwise is.

python
class Name:
    def __iter__(self):
        ...
        return an_iterator

    def __next__(self):
        ...
        return value  # or: raise StopIteration

What we're doing: Confirm that a hand-written class and a generator function produce objects that satisfy the exact same protocol, interchangeably.

count_up.pypython
class CountUp:
    def __init__(self, stop):
        self.stop = stop

    def __iter__(self):
        return CountUpIterator(self.stop)


class CountUpIterator:
    def __init__(self, stop):
        self.stop, self.current = stop, 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        self.current += 1
        return self.current


def count_up_gen(stop):
    current = 0
    while current < stop:
        current += 1
        yield current


print(list(CountUp(3)))
print(list(count_up_gen(3)))

g = count_up_gen(3)
print(hasattr(g, '__iter__'), hasattr(g, '__next__'))
print(iter(g) is g)
1
CountUp is the hand-written iterable: __iter__ builds a separate iterator object, following the split shown in the iterable-vs-iterator concept.
17
count_up_gen contains yield, so calling it never runs the body — it returns a generator object that already satisfies __iter__ and __next__.
23
Both list() calls consume something satisfying the exact same two-method protocol, despite one being 12 lines of hand-written dunders and the other being 4 lines with yield.
27
A generator object passes both hasattr checks and is its own iterator (iter(g) is g) — proof it is a full protocol implementer, not a special case a for loop treats differently.
Output
[1, 2, 3]
[1, 2, 3]
True True
True

Why this works: A for loop, list(), and every other consumer of iterables never check what class an object is — they only ever call iter() then next() and react to StopIteration. Because a generator function produces something that answers those two calls correctly, it is a complete, protocol-compliant iterator with no hand-written __iter__/__next__ at all. This is why generator functions exist: they are a shortcut for writing the protocol, not a different mechanism from it.

Assuming every iterable has a length or supports indexing

Wrong

python
def process(iterable):
    for i in range(len(iterable)):   # assumes __len__ and __getitem__
        print(iterable[i])

process(x for x in range(3))   # generator has neither

Better

python
def process(iterable):
    for item in iterable:            # only relies on the iterator protocol
        print(item)

process(x for x in range(3))

What you see: TypeError: object of type 'generator' has no len(). The fixed version prints 0, 1, 2.

Why: The iterator protocol guarantees __iter__ and __next__ only — nothing about length or indexing. A function that assumes len() or subscripting works on "any iterable" is really assuming a sequence, a narrower and stronger contract that a generator does not satisfy. Iterating with a plain for loop relies on nothing more than the protocol every iterable actually promises.

Two ways to satisfy the same protocol

Two ways to satisfy the same protocol
ApproachHow __iter__/__next__ get implemented
A class with dundersWritten by hand — __iter__ returns self or a helper object; __next__ advances state and raises StopIteration
A generator functionImplemented automatically the moment the function body contains yield

Together

python
class CountUpIterator:
    def __init__(self, stop):
        self.stop, self.current = stop, 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        self.current += 1
        return self.current

def count_up_gen(stop):
    current = 0
    while current < stop:
        current += 1
        yield current

print(list(CountUpIterator(3)))  # [1, 2, 3] - hand-written protocol
print(list(count_up_gen(3)))     # [1, 2, 3] - same protocol, generated automatically

Remember: The protocol is __iter__ + __next__ — satisfied by hand-written dunders, or automatically by any function containing yield.

See also: iterable vs iterator · generator functions · iterator protocol dunders · iter next

Advertisement

Generator functions

yield turns an ordinary function into a paused, resumable one — and yield from delegates that pause-and-resume to another iterable.

Generator functions and yield

coreintermediate

A function with yield anywhere inside it is a generator function. Calling it does not run the body — it returns a generator object that runs from the top up to the next yield each time you ask it for a value.

Think of it as

A generator function is a function that can be paused and resumed, mid-execution, as many times as it has yield statements. Calling it just builds the paused-and-ready generator; the body only actually runs in bursts, each burst ending at the next yield, driven entirely by whoever calls next().

python
def name(args):
    ...
    yield value          # pauses here, hands back value
    ...                  # resumes here on the next next() call
    # falling off the end (or a bare "return") raises StopIteration

What we're doing: Write a generator function that yields fixed-size batches from a range, and watch the body pause and resume between next() calls.

read_batches.pypython
def read_batches(total_items, batch_size):
    start = 0
    while start < total_items:
        end = min(start + batch_size, total_items)
        print(f"  computing batch [{start}, {end})")
        yield list(range(start, end))
        start = end


batches = read_batches(7, 3)
print(type(batches))
print(next(batches))
print(next(batches))
print(next(batches))
try:
    next(batches)
except StopIteration:
    print("StopIteration raised - generator exhausted")
9
Calling read_batches(7, 3) does not print anything — the body has not started. It only builds a generator object.
10
The type is "generator", not a list or any custom class — this object already satisfies the iterator protocol.
11
The first next() runs the body from the top through the print() and the yield, then pauses exactly at that yield.
12
The second next() resumes right after the yield — start = end runs, the while condition is re-checked, and it runs to the next yield.
15
After three batches, start >= total_items, the while loop exits, the function falls off the end, and that raises StopIteration.
Output
<class 'generator'>
  computing batch [0, 3)
[0, 1, 2]
  computing batch [3, 6)
[3, 4, 5]
  computing batch [6, 7)
[6]
StopIteration raised - generator exhausted

Why this works: yield is what turns a normal function definition into a generator function at compile time — Python decides this by scanning the function body once, not by anything about how it is called. Each next() resumes execution exactly where the last yield left off, with every local variable (start, end) intact, which is why the batches pick up where they left off instead of restarting from start = 0.

Writing a generator function when an eager list was actually wanted immediately

Wrong

python
def get_batches(total_items, batch_size):
    start = 0
    while start < total_items:
        end = min(start + batch_size, total_items)
        yield list(range(start, end))
        start = end

result = get_batches(7, 3)
print(len(result))   # need the count right now, before consuming anything

Better

python
def get_batches(total_items, batch_size):
    start = 0
    batches = []
    while start < total_items:
        end = min(start + batch_size, total_items)
        batches.append(list(range(start, end)))
        start = end
    return batches

result = get_batches(7, 3)
print(len(result))   # 3 - a real list has a length

What you see: TypeError: object of type 'generator' has no len()

Why: A generator object never knows its own final length in advance — it has not run far enough to know, and may never finish (an infinite generator has no length at all). If the caller genuinely needs len(), indexing, or to iterate more than once, that is a sign a plain function building a list is the right tool, not a generator function.

Calling a generator function vs. calling next()

Call the function

body does NOT run yet

Returns a generator

paused before the first line

next() runs to yield

pauses there, returns the value

Body ends

raises StopIteration

  1. Call the function — body does NOT run yet
  2. Returns a generator — paused before the first line
  3. next() runs to yield — pauses there, returns the value
  4. Body ends — raises StopIteration

Generator function vs. a normal function that builds a list

Generator function vs. a normal function that builds a list
QuestionNormal function returning a listGenerator function with yield
When does the body run?Entirely, on the callIn bursts, one per next() call
What does calling it return?The finished listA generator object, unstarted
Memory for N itemsO(N) — every item exists at onceO(1) — one item exists at a time
Can you consume it twice?Yes — the list still existsNo — exhausted after one full pass

Together

python
def squares_list(n):
    return [i * i for i in range(n)]

def squares_gen(n):
    for i in range(n):
        yield i * i

squares_list(3)        # [0, 1, 4] - built immediately, all at once
gen = squares_gen(3)    # <generator object ...> - nothing computed yet
next(gen)                # 0 - runs up to the first yield

Remember: A function with yield returns a generator when called. Its body runs in bursts, one per next(), pausing at each yield.

See also: iterator protocol · yield from · lazy evaluation · generator expressions

yield from

standardintermediate

yield from sub_iterable yields every item of sub_iterable, in order, as though the outer generator looped and yielded each one itself. It also forwards send()/throw()/close() to the sub-generator.

Think of it as

yield from hands the microphone to another generator. The outer generator steps back, the inner one speaks directly to whoever is calling next(), and control returns to the outer generator only once the inner one finishes.

python
def outer():
    yield "before"
    yield from inner_iterable   # yields every item of inner_iterable here
    yield "after"

What we're doing: Flatten an arbitrarily nested list using yield from to delegate into a recursive call, instead of a manual inner loop.

flatten.pypython
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item


result = list(flatten([1, [2, 3, [4, 5]], 6]))
print(result)


def chain_sources(*iterables):
    for iterable in iterables:
        yield from iterable


print(list(chain_sources([1, 2], (3, 4), "ab")))
4
yield from flatten(item) delegates to a fresh recursive call — every value THAT call yields comes out of the outer generator too, at whatever nesting depth.
5
A non-list item is yielded directly — the base case that stops the recursion from delegating further.
13
chain_sources delegates to each argument in turn — a list, a tuple, and a string all work, since yield from accepts any iterable, not just other generators.
Output
[1, 2, 3, 4, 5, 6]
[1, 2, 3, 4, 'a', 'b']

Why this works: Each yield from pauses the delegating generator entirely and lets the sub-iterable produce values as though there were no delegation at all — the caller of flatten() sees one flat stream of items, with no visible seam between recursive calls. Writing this with a manual inner loop (for x in flatten(item): yield x) gives the same result but does not forward send()/throw()/close() the way yield from does.

Writing the manual loop instead of yield from, and losing send()/throw() forwarding

Wrong

python
def outer_delegates(sub):
    yield "start"
    for value in sub:          # a plain loop over the sub-generator
        yield value
    yield "end"

def inner_gen():
    received = yield "ready"
    yield f"got {received}"

o = outer_delegates(inner_gen())
next(o)          # 'start'
next(o)           # 'ready'
o.send("hi")       # TypeError - "hi" goes to outer_delegates, not inner_gen

Better

python
def outer_delegates(sub):
    yield "start"
    yield from sub               # forwards send() through to sub
    yield "end"

def inner_gen():
    received = yield "ready"
    yield f"got {received}"

o = outer_delegates(inner_gen())
print(next(o))          # 'start'
print(next(o))           # delegates in, gets 'ready'
print(o.send("hi"))       # forwarded to inner_gen: 'got hi'

What you see: TypeError: can't send non-None value to a just-started generator — send() on outer_delegates lands on the plain for loop, which has no yield expression waiting to receive it.

Why: A plain for value in sub: yield value only reads values OUT of sub — it has no connection for a value sent back IN. yield from is not just shorter, it opens a two-way channel: send() and throw() called on the outer generator are forwarded straight through to whichever sub-generator is currently active behind the yield from.

Remember: yield from sub delegates the entire iteration to sub, and send()/throw()/close() pass through to it too.

See also: generator functions · sending values into generators · generator pipelines

Advertisement

Using generators well

When laziness earns its keep, how to chain generators without losing it, and the two-way channel a generator supports beyond plain iteration.

Lazy evaluation

coreintermediate

Lazy evaluation means a value is computed only when something asks for it, not when the expression is written. A generator expression is lazy; a list comprehension is eager.

Think of it as

Eager is a shopping list where every item is bought and bagged before you leave the store. Lazy is a delivery service — nothing is picked until you actually ask for the next box, and the store never holds more than the one box in transit.

python
eager = [transform(x) for x in data]   # every item computed on this line
lazy = (transform(x) for x in data)      # nothing computed until iterated

What we're doing: Run the roadmap's own list-comprehension-vs-generator-expression pair with a transform() that prints, to see exactly when each version actually runs its work.

lazy_vs_eager.pypython
def transform(x):
    print(f"  transform({x})")
    return x * x


data = [1, 2, 3]

print("building list comprehension:")
result_list = [transform(x) for x in data]
print("  list built:", result_list)

print("building generator expression:")
result_gen = (transform(x) for x in data)
print("  generator built (no transform() calls above)")
print("  now consuming with list():", list(result_gen))
9
The list comprehension runs transform() three times, once per item, before this line finishes — result_list is a complete, ordinary list by the time it is assigned.
13
The generator expression runs transform() zero times here. result_gen is only a paused generator that remembers the loop, not any computed value.
15
transform() finally runs, three times, only now — inside list(), which is the first thing that actually asks the generator for its items.
Output
building list comprehension:
  transform(1)
  transform(2)
  transform(3)
  list built: [1, 4, 9]
building generator expression:
  generator built (no transform() calls above)
  transform(1)
  transform(2)
  transform(3)
  now consuming with list(): [1, 4, 9]

Why this works: Square brackets tell Python to run the whole loop right now and collect a list; round brackets tell it to remember the loop as an unstarted plan. Both eventually call transform() the same three times and produce the same values — the difference is entirely about WHEN that work happens, which only becomes visible with a side effect like print() inside the expression.

Calling the side-effecting function again inside the generator's own filter clause

Wrong

python
def get_price(item_id):
    print(f"  fetching price for {item_id}")
    return {"a": 10, "b": 20, "c": 30}[item_id]

item_ids = ["a", "b", "c"]
cheap = (get_price(i) for i in item_ids if get_price(i) < 25)
print(next(cheap))   # fetches "a" TWICE before returning it

Better

python
def get_price(item_id):
    print(f"  fetching price for {item_id}")
    return {"a": 10, "b": 20, "c": 30}[item_id]

def priced_items(item_ids):
    for item_id in item_ids:
        price = get_price(item_id)
        if price < 25:
            yield price

item_ids = ["a", "b", "c"]
cheap = priced_items(item_ids)
print(next(cheap))   # fetches "a" once

What you see: The output shows "fetching price for a" printed twice before the first result comes back — the expensive call ran twice for the same item.

Why: Laziness controls WHEN code runs, not how many times a repeated call inside one expression runs. Writing get_price(i) once in the filter (if get_price(i) < 25) and again in the output expression calls it twice per item that passes. A generator function that computes the value once, stores it in a local variable, and yields that variable avoids the duplicate call entirely — laziness is not a substitute for computing something only once.

When transform() actually runs

List comprehension

runs transform() for every item now

Generator expression

stores the loop, runs nothing yet

Iterated later

transform() runs one item at a time

  1. List comprehension — runs transform() for every item now
  2. Generator expression — stores the loop, runs nothing yet
  3. Iterated later — transform() runs one item at a time

Eager vs lazy, side by side

Eager vs lazy, side by side
Questionresult = [transform(x) for x in data]result = (transform(x) for x in data)
When does transform() run?Immediately, for every x, on this lineOnly when result is iterated, one x at a time
What is result right after?A full list of resultsA generator object — nothing computed yet
Can it be consumed twice?YesNo — one-shot, exhausted after a full pass
Memory for a million itemsO(n) — every result storedO(1) — one result in flight

Together

python
def transform(x):
    print(f"transform({x})")
    return x * x

data = [1, 2, 3]
eager = [transform(x) for x in data]     # prints 3 lines right here
lazy = (transform(x) for x in data)       # prints nothing yet
list(lazy)                                # NOW prints 3 lines

Remember: Eager (list comprehension) computes everything now. Lazy (generator expression) computes nothing until iterated, one item at a time.

See also: generator expressions · list comprehensions · memory efficient processing · generator functions

Generator pipelines

standardintermediate

A generator pipeline chains several generator functions, each wrapping the one before it. An item flows through every stage before the next item enters the pipeline at all — no stage waits for the whole source to finish first.

Think of it as

An eager pipeline is an assembly line where each station finishes every unit before the next station starts. A generator pipeline is a bucket brigade: one item moves through every stage in turn, then the next item starts, so no stage is ever holding more than one item at a time.

python
def stage_one(source):
    for item in source:
        yield transform(item)

def stage_two(items):
    for item in items:
        if condition(item):
            yield item

pipeline = stage_two(stage_one(raw_source))   # nothing runs until iterated

What we're doing: Build a three-stage pipeline — read lines, parse them as integers, keep only the even ones — and confirm it is still one unstarted generator until iterated.

pipeline.pypython
def read_lines(lines):
    for line in lines:
        yield line


def parse_ints(lines):
    for line in lines:
        yield int(line)


def filter_even(nums):
    for n in nums:
        if n % 2 == 0:
            yield n


raw_lines = ["1", "2", "3", "4", "5", "6"]
pipeline = filter_even(parse_ints(read_lines(raw_lines)))
print(type(pipeline))
print(list(pipeline))
17
Nesting the three calls wires the pipeline: filter_even pulls from parse_ints, which pulls from read_lines, which pulls from raw_lines.
18
pipeline is a single generator object — nesting generator calls does not run any stage yet, exactly like a single generator function would not.
19
list(pipeline) is what actually starts the flow: each '4' or '6' travels through all three stages before the next raw line is even read.
Output
<class 'generator'>
[2, 4, 6]

Why this works: Nesting generator calls builds a chain of paused generators, each holding a reference to the one before it — none of them run until something calls next() on the outermost one, which then pulls from the one inside it, and so on down to the original list. This is the same lazy behaviour a single generator function has, just composed across several small, named stages instead of one large one.

Building each stage as a list, losing the point of a pipeline

Wrong

python
def parse_ints_eager(lines):
    return [int(line) for line in lines]

def filter_even_eager(nums):
    return [n for n in nums if n % 2 == 0]

raw = ["1", "2", "bad", "4"]
result = filter_even_eager(parse_ints_eager(raw))   # fails immediately, on line 1
print(result)

Better

python
def parse_ints_lazy(lines):
    for line in lines:
        yield int(line)

def filter_even_lazy(nums):
    for n in nums:
        if n % 2 == 0:
            yield n

raw = ["1", "2", "bad", "4"]
pipe = filter_even_lazy(parse_ints_lazy(raw))
print(next(pipe))   # 2 - works, "bad" has not been reached yet
print(next(pipe))   # raises ValueError only now, when parsing reaches "bad"

What you see: ValueError: invalid literal for int() with base 10: 'bad' — raised on the very first line of the eager version, before any result is available, even the valid ones that came before the bad item.

Why: parse_ints_eager must finish converting every line before returning anything, so one bad line anywhere in the source blocks every result, including the good ones that were already parsed. The generator pipeline yields 2 immediately and only encounters "bad" when the caller actually asks for the next item — valid results are usable as soon as they are produced, and a bad item is a problem only when the pipeline actually reaches it.

Remember: Chain generator functions by nesting calls. An item moves through every stage before the next item enters the pipeline.

See also: generator functions · yield from · memory efficient processing

Memory-efficient processing

standardintermediate

Processing data with generators keeps memory flat: one item exists at a time, no matter how many items the source has. Building intermediate lists instead means every item exists at once, which does not scale to a large or unbounded source.

Think of it as

A generator pipeline processes a river one cup at a time; a list-based pipeline first dams the whole river to measure it, then processes the reservoir. The cup approach works on a river of any size, including one whose end you cannot see. The dam approach needs somewhere to put all that water first.

python
# grows with the source
rows = [transform(line) for line in open(path)]

# flat, regardless of source size
rows = (transform(line) for line in open(path))
for row in rows:
    handle(row)

What we're doing: Compare the memory a list and a generator hold for the same 500,000-item source, and confirm the generator still visits every item correctly.

memory_compare.pypython
import sys

N = 500_000
list_version = [f"row-{i}" for i in range(N)]
gen_version = (f"row-{i}" for i in range(N))

print("list size:", sys.getsizeof(list_version))
print("gen size:", sys.getsizeof(gen_version))
print("count via generator:", sum(1 for _ in gen_version))
4
list_version builds and stores half a million strings before this line finishes.
5
gen_version stores only the loop and range(N) — not one string exists yet.
9
Consuming gen_version with sum() still visits all 500,000 items — laziness changes when the work happens, never whether it happens.
Output
list size: 4167352
gen size: 208
count via generator: 500000

Why this works: sys.getsizeof reports the size of the object itself, not what it will eventually produce — a generator's size is the same small constant whether it has one item or a billion, because it holds a paused loop, not results. The list's size scales directly with N because every item is materialized and kept.

Chaining comprehensions with square brackets, silently reintroducing full materialization

Wrong

python
def process_large_file_eager(lines):
    cleaned = [line.strip() for line in lines]      # full list #1
    non_empty = [line for line in cleaned if line]   # full list #2
    return non_empty

Better

python
def process_large_file_lazy(lines):
    cleaned = (line.strip() for line in lines)       # generator, no list
    non_empty = (line for line in cleaned if line)    # generator, no list
    return non_empty

What you see: On a file with millions of lines, the eager version holds two full-length lists in memory at once (cleaned and non_empty) even though only non_empty is ever returned — a memory spike with no matching benefit.

Why: Each square-bracket comprehension in the chain fully materializes before the next line runs, so intermediate lists pile up even when nothing needs them to persist. Swapping every intermediate comprehension for a generator expression keeps each stage lazy, so the whole chain processes one line at a time end to end — matching what `process_large_file_eager` and `process_large_file_lazy` actually produce, which is identical for well-formed input.

Remember: A pipeline built entirely of generators keeps memory flat. Converting any stage to a list brings the whole source into memory at that point.

See also: lazy evaluation · generator pipelines · generator expressions · generators and iterators for memory

Sending values into generators

standardadvanced

gen.send(value) resumes a paused generator and delivers value as the result of the yield expression it paused on. next(gen) is exactly gen.send(None).

Think of it as

next() only ever pulls a value out. send() does the same pull, but also drops a value into the generator's hands on the way in — value = yield captures whatever send() delivers into a real local variable, resuming execution right after that line.

python
def name():
    received = yield initial_value   # pauses here, returns initial_value
    ...                                # resumes here, received == whatever send() delivered

gen = name()
first = next(gen)          # prime it - runs to the first yield
result = gen.send(value)    # delivers value, resumes, runs to the next yield

What we're doing: Build a generator that keeps a running average, primed with next() and fed new numbers with send(), and show real values going in and real averages coming out.

running_average.pypython
def running_average():
    total = 0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count


avg_gen = running_average()
print(next(avg_gen))          # prime it - runs to the first yield, returns None
print(avg_gen.send(10))       # delivers 10, resumes, yields the new average
print(avg_gen.send(20))
print(avg_gen.send(30))
6
value = yield average pauses here on the first next(), handing out None (the initial average) and waiting for a value to be sent in.
12
next(avg_gen) is the required priming call — it runs the body up to the first yield and returns average, which is still None; nothing has been sent yet.
13
send(10) delivers 10 into the paused yield expression, so value becomes 10, the running total and count update, and execution pauses again at the next yield, handing back the new average.
Output
None
10.0
15.0
20.0

Why this works: yield is an expression that can produce a value out AND receive a value in, at the same pause point. Priming with next() is required because the first yield has not been reached yet — there is nowhere for a sent value to go until execution has paused on a yield expression waiting to be assigned.

Calling send() with a real value before priming the generator

Wrong

python
def broken_average():
    total = 0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

gen = broken_average()
gen.send(10)   # sent before the generator has reached its first yield

Better

python
def broken_average():
    total = 0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

gen = broken_average()
next(gen)          # prime it first - required before any real send()
print(gen.send(10))

What you see: TypeError: can't send non-None value to a just-started generator

Why: A freshly created generator has not executed a single line yet — there is no yield expression currently paused to receive a value. gen.send(None) (equivalently, next(gen)) is the only thing that can legally start it, running to the first yield; only after that can a real value be sent in.

Remember: send(value) resumes a generator, making value the result of the yield it paused on. Prime with next() first.

See also: generator functions · yield from · generator cleanup

Generator cleanup

standardadvanced

gen.close() raises GeneratorExit at the generator's current paused point, which a try/finally block can catch to release a resource. After close(), the generator is permanently exhausted — a later next() raises StopIteration.

Think of it as

A generator paused mid-loop is like a worker on a coffee break with a resource checked out. close() is the manager walking over and ending the shift right there — a finally block is the worker's standing instruction to return anything checked out before actually leaving, no matter when the shift ends.

python
def managed():
    try:
        while True:
            yield "data"
    finally:
        release_resource()   # runs on close(), on completion, or on garbage collection

gen = managed()
next(gen)
gen.close()   # raises GeneratorExit inside managed(), runs the finally block

What we're doing: Show close() actually raising GeneratorExit inside a paused generator, running its finally block, and leaving the generator exhausted afterward.

managed_resource.pypython
def managed_resource():
    print("  resource opened")
    try:
        while True:
            yield "data"
    finally:
        print("  resource closed")


gen = managed_resource()
print(next(gen))
print(next(gen))
gen.close()
try:
    next(gen)
except StopIteration:
    print("generator exhausted after close()")
10
The first next() runs the body up to the first yield, printing "resource opened" along the way.
12
close() raises GeneratorExit right at the paused yield inside the while loop — the finally block catches that unwind and prints "resource closed".
14
Calling next() again on the now-closed generator raises StopIteration, not GeneratorExit — close() leaves the generator permanently exhausted, the same state a normal completion leaves it in.
Output
  resource opened
data
data
  resource closed
generator exhausted after close()

Why this works: close() does not simply stop calling next() — it throws GeneratorExit into the generator's exact paused location, giving any try/finally a chance to run exactly as it would for a real exception. Once that unwind finishes, the generator is done: it cannot be resumed, only re-created.

Catching GeneratorExit and yielding again instead of letting it propagate

Wrong

python
def bad_cleanup():
    try:
        while True:
            yield "x"
    except GeneratorExit:
        yield "one more"   # illegal - yielding in response to close()

gen = bad_cleanup()
next(gen)
gen.close()

Better

python
def good_cleanup():
    try:
        while True:
            yield "x"
    except GeneratorExit:
        print("cleaning up, not yielding")
        raise   # re-raise - lets the generator actually stop

gen = good_cleanup()
next(gen)
gen.close()

What you see: RuntimeError: generator ignored GeneratorExit — close() fails loudly instead of the generator quietly continuing to produce values.

Why: GeneratorExit exists specifically to stop a generator — Python treats a generator that responds to it by yielding again as broken, since that generator is refusing to actually close. A handler that needs to run cleanup code should do so and then either let GeneratorExit propagate (often by simply not catching it, or by re-raising) rather than yielding.

Remember: close() raises GeneratorExit at the generator's paused point — a finally block runs, then it is exhausted for good.

See also: generator functions · sending values into generators · iterator protocol dunders

Advertisement