beginnerintermediateadvanced

Python · Section 4

Functional Programming

Level: intermediate
Read: 58 min
Concepts: 8

Python uses functional concepts heavily without being a functional language: functions are ordinary objects, map/filter/reduce transform data without explicit loops, and immutable data shares safely across a program. This section covers what is not already taught in Functions — purity, composition, folding, functools, laziness, and immutability as a design choice — and points back to first-class functions, higher-order functions, closures, lambda, map, and filter where the roadmap repeats itself.

This section

Functional Programming in Python is AwesomeIndently

What is true here

A pure function reads only its arguments and returns only a value — no global, no file, no mutated argument.
functools.reduce(func, iterable, initial) folds an iterable to one value, eagerly, left to right.
functools.partial(func, *args, **kwargs) pre-binds arguments into a new, smaller-arity callable.
map(), filter(), and every generator are lazy — nothing computes until something iterates.
tuple, frozenset, namedtuple, and a frozen dataclass all refuse mutation after creation.

What you will be able to do

Tell a pure function from an impure one, and explain why purity makes testing and caching easier
Compose several small functions into one transformation, by hand or with reduce()
Fold a list into one accumulated value with functools.reduce, and know when sum()/a comprehension reads better
Pre-bind arguments with functools.partial instead of writing a throwaway wrapper function
Explain why a generator pipeline holds flat memory, and prove a lazy expression has not run yet
Choose an immutable data structure (tuple, frozenset, namedtuple, frozen dataclass) when a value needs to be shared safely

A functional-style transformation, end to end

An iterable source

a file, a list, an API page

Composed pure functions

each stage transforms, none mutate or reach outside

Lazy generator pipeline

nothing computed until pulled, one item at a time

reduce() folds it down

many items become one accumulated result

Purity and composition

What makes a function safe to reason about in isolation, and how small functions combine into a bigger transformation. First-class functions, higher-order functions, closures, and lambda — this neighborhood's other four roadmap items — are taught in Functions; see the linked concepts below.

Pure functions

coreintermediate

A pure function returns a value that depends only on its arguments, and changes nothing outside itself — no global variable, no file, no argument mutated in place. Call it twice with the same input and it gives the same answer, every time.

Think of it as

A vending machine, not a bank teller. Put in the same coins and the same button press, and the same snack drops out every time — it never remembers your last visit, and pressing the button never changes what is behind any other button.

python

def add_tax(price, rate):        # pure: output depends only on price and rate
    return price * (1 + rate)

add_tax(100, 0.08)               # 108.0
add_tax(100, 0.08)               # 108.0 — identical call, identical result

What we're doing: Compare a pure tax calculation against an impure one that accumulates a running total in a global, and show how the impure version breaks "same input, same output."

purity.pypython

def add_tax_pure(price, rate):
    return price * (1 + rate)


total = 0


def add_tax_impure(price, rate):
    global total
    total += price * rate
    return price * (1 + rate)


print(add_tax_pure(100, 0.08))
print(add_tax_pure(100, 0.08))

print(add_tax_impure(100, 0.08))
print(total)
print(add_tax_impure(100, 0.08))
print(total)

2: add_tax_pure reads only price and rate, its own arguments — nothing else exists in its body.
14: Called twice with the same arguments, add_tax_pure returns the identical 108.0 both times.
9: global total lets add_tax_impure both read and reassign a variable outside its own scope.
10: This line is the side effect: every call quietly grows total, a piece of state the caller never asked for.
18: total is 8.0 after one call — evidence that add_tax_impure changed something beyond its return value.
19: The same call again returns the same 108.0, but total has now doubled to 16.0 — the hidden state kept moving.

Output

Why this works: add_tax_pure's entire behavior is determined by price and rate — there is nothing else in its body that could make one call differ from an identical one, so it is safe to call repeatedly, cache, or run in parallel without surprises. add_tax_impure returns the correct tax amount every time too, but it also silently grows total on every call — a second, hidden output no caller asked for and no return value reveals. That gap is exactly what 'pure' rules out: a pure function's contract is fully described by its parameters and its return value, an impure one's is not.

Assuming a function is pure because it "just computes something"

Wrong

python

def add_item(cart, item):
    cart.append(item)
    return cart

Better

python

def add_item(cart, item):
    return cart + [item]

What you see: cart1 = add_item(cart, 'pen') looks like it produces a new cart, but the caller's original cart list has also silently grown by one item — checked with cart is cart1, which is True.

Why: cart.append(item) mutates the list the caller passed in, in place — the return value is not a new object, just the same list handed back. That makes add_item impure: calling it changes something the caller can observe beyond the return value. cart + [item] builds and returns a brand-new list instead, leaving the original untouched — the fix that actually makes the function pure.

Pure vs impure, same job

Pure

+Reads only its own arguments
+Returns a value, changes nothing else
+Same input always gives the same output

Impure

−Reads or writes state outside itself
−A global here accumulates across calls
−Same input can give a different output

What makes a function impure

What makes a function impure
Impure behavior	Why it breaks purity
Reads or writes a global variable	Output depends on, or changes, state outside the call
Mutates a list or dict argument	The caller sees a side effect, not just a return value
Calls print(), writes a file, hits the network	Affects the outside world beyond the return value
Calls random.random() or datetime.now()	Same arguments give a different result on each call

Together

python

def apply_discount(price, rate):        # pure — only reads its own arguments
    return price * (1 - rate)

def apply_discount_logged(price, rate, log):  # impure — mutates a shared list
    result = price * (1 - rate)
    log.append(result)
    return result

Remember: A pure function only reads its arguments and only returns a value — no global, no file, no mutated argument. Same input always means same output.

Function composition

standardintermediate

Composing functions means feeding one function's output straight into the next, building a bigger transformation from small pure ones. Python has no built-in compose() — the pattern is written by hand, usually with a small helper.

Think of it as

An assembly line, not a single machine. Each station does one small, well-defined job and passes its output to the next station untouched — the finished product is the sum of every station, but no single station needs to know about the others.

python

def compose(f, g):
    return lambda x: f(g(x))     # g runs first, then f on its result

shout = compose(str.upper, str.strip)
shout("  hello  ")                # 'HELLO'

What we're doing: Compose two functions with a two-argument helper, then compose an arbitrary number with reduce().

compose_demo.pypython

def compose(f, g):
    return lambda x: f(g(x))


def strip_spaces(s):
    return s.strip()


def to_upper(s):
    return s.upper()


shout = compose(to_upper, strip_spaces)
print(shout("  hello  "))

from functools import reduce


def compose_many(*funcs):
    return reduce(lambda f, g: lambda x: g(f(x)), funcs)


pipeline = compose_many(strip_spaces, to_upper, lambda s: s + "!")
print(pipeline("  hi  "))

2: compose(f, g) returns a new function that calls g first, then passes its result into f.
13: shout runs strip_spaces first (removing the outer spaces), then to_upper on what strip_spaces returned.
20: compose_many folds any number of functions with reduce, each stage taking the previous result as input.
24: The pipeline reads left to right here — strip, then upper, then append "!" — the opposite order from a nested compose(f, g) call.

Output

HELLO
HI!

Why this works: compose(f, g) builds and returns a lambda that calls g(x) first and hands the result to f — nothing runs until the returned function is actually called, exactly like any other closure. compose_many uses reduce to fold a list of functions into one function step by step, each new lambda wrapping the one before it, so calling the final result runs every original function in the order they were listed.

Getting f(g(x)) backwards — the wrong function runs first

Wrong

python

def compose(f, g):
    return lambda x: f(g(x))

parse_then_double = compose(str.strip, lambda n: n * 2)
parse_then_double(5)

Better

python

def compose(f, g):
    return lambda x: f(g(x))

double_then_str = compose(str, lambda n: n * 2)
double_then_str(5)

What you see: AttributeError: 'int' object has no attribute 'strip' — g(x) runs first (doubling 5 to 10), then f tries to call .strip() on the resulting int.

Why: compose(f, g)(x) always runs g first and feeds its output into f — the rightmost function in the call is the first one that actually runs. str.strip expects a string, but doubling 5 produces the int 10, so the composition fails the moment f receives a type g was never meant to hand it. Composed functions must agree at their boundary: each one's return type has to match what the next one in the chain expects as input.

Composing two functions vs several

Composing two functions vs several
Goal	Shape
Compose exactly two	compose(f, g) = lambda x: f(g(x))
Compose any number, left to right	reduce(lambda f, g: lambda x: g(f(x)), funcs)
Read as a pipeline instead	value = f(g(h(value))) — read right to left

Together

python

def compose(f, g):
    return lambda x: f(g(x))

shout = compose(str.upper, str.strip)
shout("  hello  ")   # 'HELLO'

Remember: compose(f, g)(x) runs g first, then f on its result. Python has no built-in compose — reduce() folds any number of functions into one.

reduce and functools

Folding an iterable to one value, and the standard-library module that tool lives in.

reduce

coreintermediate

functools.reduce(func, iterable, initial) folds an iterable into one value: it calls func(accumulator, item) for each item, left to right, carrying the result forward. Without initial, the first item becomes the starting accumulator.

Think of it as

A snowball rolled down a hill. Each item it passes over sticks to the ball, so what comes out the bottom is one object built from every item along the way — never a separate result per item, the way map produces one.

python

from functools import reduce

reduce(func, iterable)             # func(func(item1, item2), item3), ... — starts from item1
reduce(func, iterable, initial)    # func(initial, item1), then item2, ... — starts from initial

What we're doing: Sum a list with and without an explicit initial value, confirm reduce runs immediately (not lazily), and fold a list of words into a frequency dict.

reduce_demo.pypython

from functools import reduce

nums = [1, 2, 3, 4]
print(reduce(lambda acc, n: acc + n, nums))
print(reduce(lambda acc, n: acc + n, nums, 100))
print(reduce(lambda acc, n: acc * n, nums, 1))

try:
    reduce(lambda acc, n: acc + n, [])
except TypeError as e:
    print("TypeError:", e)
print(reduce(lambda acc, n: acc + n, [], 0))

words = ["apple", "banana", "apple", "cherry", "banana", "apple"]
counts = reduce(lambda acc, w: {**acc, w: acc.get(w, 0) + 1}, words, {})
print(counts)

4: No initial value — reduce starts with 1 as the accumulator and folds in 2, then 3, then 4.
5: initial=100 changes the starting accumulator; the fold itself works the same way from there.
6: A running product instead of a sum — reduce does not care what func does, only that it takes two arguments.
9: An empty iterable with no initial has nothing to seed the accumulator with, so reduce raises immediately.
12: Supplying initial=0 covers the empty case — the loop body just never runs, and 0 comes back unchanged.
15: Folding into a dict works the same way as folding into a number — acc is just a different type, rebuilt each step.

Output

10
110
24
TypeError: reduce() of empty iterable with no initial value
0
{'apple': 3, 'banana': 2, 'cherry': 1}

Why this works: reduce(func, iterable, initial) walks the iterable exactly once, left to right, calling func(accumulator, item) and replacing the accumulator with whatever func returns each time — the value returned after the last item is reduce's own return value. With no initial, the first item is used as the starting accumulator instead of a supplied one, which is also why an empty iterable with no initial has nothing to start from and raises TypeError rather than silently returning something. Nothing about reduce is lazy: unlike map or filter, it must exhaust the whole iterable before it can return anything, since the final value depends on every item having been folded in.

Reaching for reduce() where sum(), any(), or a comprehension already say it more clearly

Wrong

python

from functools import reduce

nums = [1, 2, 3, 4]
total = reduce(lambda acc, n: acc + n, nums, 0)
doubled = reduce(lambda acc, n: acc + [n * 2], nums, [])

Better

python

nums = [1, 2, 3, 4]
total = sum(nums)
doubled = [n * 2 for n in nums]

What you see: Both versions produce the same 10 and [2, 4, 6, 8] — reduce is not wrong here, just harder to read than the tool already built for the job.

Why: sum() exists specifically for the summing case, and a list comprehension already reads as "build a new list from this one" without a lambda or an accumulator to track — both are the idiomatic Python for their job. reduce earns its place for a genuinely custom fold with no dedicated builtin: building a frequency dict, chaining function composition, or any accumulation whose shape is not "add up the numbers" or "transform each item."

One accumulator, carried through every item

[1, 2, 3, 4]

the iterable, left to right

func(acc, item)

called once per item, accumulator carried forward

one final value, not one per item

reduce() with and without an initial value

reduce() with and without an initial value
Call	Result
reduce(lambda acc, n: acc + n, [1, 2, 3, 4])	10 — starts from the first item, 1
reduce(lambda acc, n: acc + n, [1, 2, 3, 4], 100)	110 — starts from 100 instead
reduce(lambda acc, n: acc * n, [1, 2, 3, 4], 1)	24 — a running product
reduce(lambda acc, n: acc + n, [])	TypeError — nothing to start from
reduce(lambda acc, n: acc + n, [], 0)	0 — initial covers the empty case

Together

python

from functools import reduce

nums = [1, 2, 3, 4]
reduce(lambda acc, n: acc + n, nums)        # 10
reduce(lambda acc, n: acc + n, nums, 100)   # 110

Remember: reduce(func, iterable, initial) folds left to right into one value, calling func(acc, item). It always runs eagerly — never lazy, unlike map or filter.

functools

referenceintermediate

functools is the standard-library module for working with functions themselves — folding with reduce, binding arguments with partial, and caching results. reduce and partial each get their own concept; this is the rest of the module.

python

from functools import reduce, partial, lru_cache

reduce(lambda acc, n: acc + n, [1, 2, 3])   # fold to one value
double = partial(lambda a, b: a * b, 2)     # pre-bind an argument

@lru_cache(maxsize=None)                    # cache by argument
def slow(n): ...

functools at a glance

functools at a glance
Tool	What it does
reduce(func, iterable, initial)	Folds an iterable into one accumulated value
partial(func, args, *kwargs)	Returns a new callable with some arguments pre-bound
@lru_cache(maxsize=None)	Caches return values by argument, evicting the least recently used when full
@cache	Caches every return value forever — an unbounded lru_cache(maxsize=None)
@wraps(func)	Copies func's __name__, __doc__, etc. onto a wrapper — keeps decorated functions introspectable

Together

python

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

fib(20)              # 6765, computed once per unique n
fib.cache_info()      # CacheInfo(hits=18, misses=21, maxsize=None, currsize=21)

Remember: functools is where Python keeps its function-level tools — reduce, partial, and caching decorators like lru_cache and cache all live here.

functools.partial

coreintermediate

functools.partial(func, *args, **kwargs) returns a new callable with those arguments already bound. Calling the result only requires the arguments that were left unbound — the pre-bound ones are filled in automatically every time.

Think of it as

A pre-filled order form. Instead of writing the same recurring details on every request, partial fills them in once — the person submitting it only adds what actually changes each time.

python

from functools import partial

square = partial(power, exponent=2)   # exponent pre-bound by keyword
square(5)                              # 25 — only base supplied

get_request = partial(send_request, "GET")   # method pre-bound by position
get_request(url, timeout=5)                   # method already filled in

What we're doing: Pre-bind a keyword argument to build two specialized functions from one general one, then pre-bind a positional argument for an HTTP-style helper.

partial_demo.pypython

from functools import partial


def power(base, exponent):
    return base ** exponent


square = partial(power, exponent=2)
print(square(5))
cube = partial(power, exponent=3)
print(cube(2))


def send_request(method, url, timeout=30):
    return f"{method} {url} timeout={timeout}"


get_request = partial(send_request, "GET")
print(get_request("https://api.example.com/users"))
print(get_request("https://api.example.com/users", timeout=5))

7: partial(power, exponent=2) returns a new callable — exponent is already 2 every time it is called.
8: square(5) only supplies base — exponent was bound when square was built, not here.
9: A second, independent partial — cube binds exponent to 3 instead, unrelated to square.
18: method is bound positionally here, as "GET" — every call through get_request already has it filled in.
20: A keyword argument left unbound by the partial can still be overridden per call, exactly like calling send_request directly.

Output

25
8
GET https://api.example.com/users timeout=30
GET https://api.example.com/users timeout=5

Why this works: partial(func, *args, **kwargs) does not call func at all — it returns a new callable object that remembers func and the arguments given, and only calls func when that new callable is itself called, merging in whatever arguments arrive at that point. square and cube are both built from the same power function but remember a different exponent, because each call to partial() creates its own independent bound-argument record, the same way two calls to a factory function build independent closures. Arguments left unbound, like timeout in get_request, still work exactly as they would calling send_request directly — a partial only fixes what it was given, nothing more.

Binding a positional argument in the wrong slot

Wrong

python

from functools import partial

def greet(greeting, name):
    return f"{greeting}, {name}!"

name_is_hello = partial(greet, "Hello")
print(name_is_hello("Kip", "extra"))

Better

python

from functools import partial

def greet(greeting, name):
    return f"{greeting}, {name}!"

hello = partial(greet, "Hello")
print(hello("Kip"))

What you see: TypeError: greet() takes 2 positional arguments but 3 were given — the partial already filled the first positional slot (greeting), so a caller supplying two more positional arguments overshoots.

Why: partial(greet, "Hello") binds "Hello" to the FIRST unbound positional parameter, greeting — it does not reserve a named slot, only a position. Every positional argument supplied at call time is appended after the ones already bound, so hello("Kip") correctly fills name, but adding a second positional argument collides with there being no third parameter left to fill. Passing extra arguments the underlying function does not accept fails exactly like calling the original function that way would.

One call becomes two smaller ones

power(base, exponent)

the original, two-argument function

partial(power, exponent=2)

exponent bound now, base still open

square(5)

only base is supplied — 25

Binding arguments with partial

Binding arguments with partial
Call	Result
partial(power, exponent=2)	a new callable — only base is still needed
square(5)	25 — 5 supplied as base, exponent=2 already bound
partial(send_request, "GET")	GET is bound as the first positional argument, method
get_request("/users")	'GET /users timeout=30' — url supplied, method already bound
p.func, p.args, p.keywords	inspect what a partial actually bound

Together

python

from functools import partial

def power(base, exponent):
    return base ** exponent

square = partial(power, exponent=2)
square(5)   # 25
square(6)   # 36

Remember: partial(func, *args, **kwargs) pre-binds arguments and returns a new callable; only the unbound arguments are needed at call time.

See also: functools module · closures · keyword arguments

Laziness and immutability

Computing only what is asked for, and choosing data that cannot be changed out from under you.

Lazy evaluation

coreintermediate

Lazy evaluation delays computing a value until something actually needs it, instead of computing it the moment the expression is written. map(), filter(), and generators are all lazy — building one does no work; iterating it does.

Think of it as

A restaurant order versus a buffet. A buffet cooks everything up front, whether or not anyone eats it; an order only gets cooked once a customer actually asks for that dish — map, filter, and generators all work the buffet's opposite way, doing nothing until asked.

python

gen = (f(x) for x in source)   # built instantly, computes nothing yet
next(gen)                       # now exactly one item is computed
list(gen)                       # drives the rest, one at a time

What we're doing: Prove laziness with an observable side effect: build a generator expression and a map object, and show neither runs its function until iteration actually pulls a value.

lazy_demo.pypython

def loud(n):
    print(f"computing {n}")
    return n * n


gen = (loud(n) for n in range(3))
print("generator built, nothing printed yet")
print(next(gen))
print(next(gen))

m = map(loud, range(3))
print("map built, nothing printed yet")
print(next(m))

6: The generator expression is built here — if it were eager, "computing 0" would already have printed.
7: This line runs and prints BEFORE any "computing" line — direct proof nothing has been computed yet.
8: next(gen) is the first thing that actually calls loud(), computing exactly one item: 0.
9: A second next() computes exactly the next one, 1 — never more than what was asked for.
11: map(loud, range(3)) is equally lazy — building it does not call loud even once.
13: Only this next() finally triggers the first call to loud, mirroring the generator expression above.

Output

generator built, nothing printed yet
computing 0
0
computing 1
1
map built, nothing printed yet
computing 0
0

Why this works: Both a generator expression and map() return an iterator immediately when constructed — an object that knows HOW to produce the next value, not one that has already produced it. "generator built, nothing printed yet" proves this: if construction ran loud(), that print would appear before the message, not after. Only next() (or anything that iterates, like a for loop or list()) actually pulls a value, which is the one moment loud() runs — and it runs exactly once per item pulled, never ahead of demand.

Assuming a lazy object has already validated or loaded its data

Wrong

python

def parse_rows(rows):
    for row in rows:
        yield int(row)

pipeline = parse_rows(["1", "2", "bad", "4"])
print("pipeline created — no error, so the data must be fine")
print(list(pipeline))

Better

python

def parse_rows(rows):
    for row in rows:
        yield int(row)

pipeline = parse_rows(["1", "2", "bad", "4"])
try:
    print(list(pipeline))
except ValueError as e:
    print(f"bad row in data: {e}")

What you see: Building pipeline raises nothing at all — the ValueError for int('bad') only appears later, when list(pipeline) actually iterates far enough to reach that row.

Why: A generator function's body does not run at all when it is called — calling parse_rows(rows) only builds a generator object; none of int(row) has executed yet, so no bad row has been touched. The error only surfaces once iteration reaches that specific item, which can be lines of code (or an entire request) away from where the generator was built. Wrapping the CONSUMING code in try/except, not the construction, is where a lazy pipeline's errors actually need to be caught.

Building vs consuming

Eager — a list comprehension

+Every item is computed the moment the line runs
+All results held in memory at once
+No further work happens on later access

Lazy — a generator expression

−Nothing computed when the line runs
−One item computed per next() or iteration step
−Never holds more than the current item

Lazy vs eager, side by side

Lazy vs eager, side by side
Expression	Lazy?
(x * x for x in nums)	yes — a generator expression
map(str.upper, names)	yes — nothing runs until consumed
filter(is_valid, rows)	yes — same as map
[x * x for x in nums]	no — a list comprehension computes everything now
sorted(x for x in nums)	the generator is lazy, but sorted() immediately consumes it all

Together

python

gen = (n * n for n in range(1_000_000))   # instant — no work done yet
next(gen)                                  # 0 — only now is one item computed

Remember: A lazy object (map, filter, a generator) does no work when built — only next() or iteration triggers computation, one item at a time.

See also: generator pipelines · generator expressions · map · filter

Generator pipelines

coreadvanced

A generator pipeline chains several generator functions, each one consuming the previous stage and yielding its own transformed items. Every item flows through every stage one at a time — no stage ever holds the whole dataset.

Think of it as

A bucket brigade, not a warehouse. Each worker (stage) passes one bucket (item) straight to the next the moment they finish with it — nobody stockpiles buckets, and the whole line can keep going as long as buckets keep arriving.

python

def stage_one(source):
    for item in source:
        yield transform(item)

def stage_two(items):
    for item in items:
        if keep(item):
            yield item

pipeline = stage_two(stage_one(raw_source))   # nested calls wire the stages

What we're doing: Wire four generator stages into one pipeline that cleans, parses and filters raw lines, and confirm each item is pulled through every stage before the next one starts.

pipeline_demo.pypython

def read_lines():
    for line in ["  10,widget", "  -5,gadget", "20,gizmo", "  "]:
        yield line


def strip_lines(lines):
    for line in lines:
        yield line.strip()


def nonempty(lines):
    for line in lines:
        if line:
            yield line


def parse(lines):
    for line in lines:
        qty_str, name = line.split(",")
        yield int(qty_str), name


def positive_only(records):
    for qty, name in records:
        if qty > 0:
            yield qty, name


pipeline = positive_only(parse(nonempty(strip_lines(read_lines()))))
print(list(pipeline))

1: read_lines is the source stage — raw, unprocessed strings, one blank and one negative quantity mixed in.
6: strip_lines only removes surrounding whitespace — it does not know or care about the later stages.
11: nonempty drops the blank line the source produced, before parse ever has to look at it.
17: parse turns a "qty,name" string into an (int, str) tuple — it assumes nonempty already removed blanks.
23: positive_only is the last stage — it drops the -5 gadget record, keeping only meaningful quantities.
27: Nesting the calls wires the stages; nothing runs until list() on the next line actually consumes the outermost one.

Output

[(10, 'widget'), (20, 'gizmo')]

Why this works: Each generator function's body does not run when it is called — calling strip_lines(read_lines()) only builds a generator object wrapping another generator object, four layers deep by the time positive_only wraps everything. list(pipeline) is what finally drives the outermost stage, which pulls one item from parse, which pulls one from nonempty, which pulls one from strip_lines, which pulls one from read_lines — one raw line travels all the way through the chain before the next raw line is even read. That is what keeps memory flat: no stage ever holds more than the one item currently passing through it, regardless of how long read_lines eventually becomes.

Breaking the pipeline by materializing a stage in the middle

Wrong

python

def strip_lines(lines):
    return [line.strip() for line in lines]   # a list, not a generator

def nonempty(lines):
    for line in lines:
        if line:
            yield line

pipeline = nonempty(strip_lines(huge_log_file))

Better

python

def strip_lines(lines):
    for line in lines:
        yield line.strip()   # still lazy — one item at a time

def nonempty(lines):
    for line in lines:
        if line:
            yield line

pipeline = nonempty(strip_lines(huge_log_file))

What you see: The wrong version works for a small file, but strip_lines([...]) reads and holds every stripped line in memory at once before nonempty even starts — for a huge_log_file too large to fit in memory, the process runs out of memory before producing a single result.

Why: A list comprehension is eager — return [line.strip() for line in lines] fully materializes the result before returning, which throws away the whole point of chaining generators: keeping only one item resident at a time. Every stage in a pipeline needs to be a generator (using yield) or another lazy construct for the flat-memory property to hold end to end — one eager stage anywhere in the chain forces everything before it to be fully computed up front.

Each stage pulls from the one before it

read_lines()

raw source lines

strip → filter → parse

each stage yields, never collects

positive_only(...)

final stage, pulled by list() or a for loop

A four-stage pipeline, each stage in one line

A four-stage pipeline, each stage in one line
Stage	Job
strip_lines(lines)	yields each line.strip()
nonempty(lines)	yields only lines that survived strip() non-empty
parse(lines)	yields (quantity, name) parsed from each "qty,name" line
positive_only(records)	yields only records where quantity > 0

Together

python

pipeline = positive_only(parse(nonempty(strip_lines(read_lines()))))
list(pipeline)   # only the valid, positive-quantity records

Remember: Nesting generator function calls builds a pipeline where each item flows through every stage before the next item starts — no intermediate list, ever.

Immutability concepts

standardintermediate

Functional-style Python favors data that cannot change after creation — tuple, frozenset, namedtuple, a frozen dataclass — so a value can be passed and shared freely, with no risk that one piece of code edits it out from under another.

Think of it as

A signed contract versus a shared whiteboard. Once signed, a contract cannot be edited — anyone holding a copy can rely on it staying exactly as agreed. A shared whiteboard can be changed by whoever has a marker, so no one holding it can fully trust it stayed the same since they last looked.

python

from collections import namedtuple
import dataclasses

Point = namedtuple("Point", ["x", "y"])   # immutable, field-named tuple

@dataclasses.dataclass(frozen=True)        # raises on attribute assignment
class Config:
    host: str
    port: int

What we're doing: Confirm four immutable structures actually refuse to change: a namedtuple field, a frozenset, a frozen dataclass field, and the boundary case of a tuple holding a mutable list.

immutability_demo.pypython

from collections import namedtuple
import dataclasses

Point = namedtuple("Point", ["x", "y"])
p1 = Point(1, 2)
try:
    p1.x = 5
except AttributeError as e:
    print("AttributeError:", e)

fs = frozenset([1, 2, 3])
try:
    fs.add(4)
except AttributeError as e:
    print("AttributeError:", e)

t = (1, 2, [3, 4])
t[2].append(5)
print(t)


@dataclasses.dataclass(frozen=True)
class Config:
    host: str
    port: int


c = Config("localhost", 8080)
try:
    c.port = 9090
except dataclasses.FrozenInstanceError as e:
    print("FrozenInstanceError:", e)

7: A namedtuple is still a tuple underneath — assigning to a field is rejected exactly like assigning to any tuple slot would be.
13: frozenset has no .add() at all — every set method that would mutate is simply absent from its type.
16: The tuple t itself never changes — but its third slot holds a real, mutable list, and that list can still be edited in place.
30: A frozen dataclass overrides attribute assignment itself, raising before the assignment can take effect.

Output

AttributeError: can't set attribute
AttributeError: 'frozenset' object has no attribute 'add'
(1, 2, [3, 4, 5])
FrozenInstanceError: cannot assign to field 'port'

Why this works: Each of these types refuses mutation through a different mechanism: namedtuple and tuple simply define no operation that reassigns a slot, frozenset omits every mutating method a plain set has, and a frozen dataclass overrides __setattr__ to raise on any assignment after __init__. The tuple-holding-a-list case is the one gap worth remembering: (1, 2, [3, 4]) guarantees the TUPLE'S three slots never point at different objects, but slot three's object is a list, and nothing about the outer tuple stops that inner list from being mutated — immutability does not recurse into what an immutable container holds unless every level is immutable too.

Treating "wrapped in a tuple" as "safe from every mutation"

Wrong

python

CACHE_KEYS = (1, 2, [3, 4])   # looks locked down — it is a tuple

def add_key(new_key):
    CACHE_KEYS[2].append(new_key)   # "safe", CACHE_KEYS is a tuple...

Better

python

CACHE_KEYS = (1, 2, (3, 4))   # tuple all the way down — genuinely immutable

def with_extra_key(new_key):
    return CACHE_KEYS[:2] + (CACHE_KEYS[2] + (new_key,),)

What you see: Every caller of add_key mutates the SAME nested list that every other holder of CACHE_KEYS sees — the module-level "constant" silently grows, with no error to signal it.

Why: CACHE_KEYS being a tuple only guarantees CACHE_KEYS itself always points at the same three slots — it makes no promise about what those slots contain. Slot two holds a list, and .append() mutates that list in place regardless of what container holds a reference to it. Nesting an immutable type (a tuple, not a list) at every level, as the fixed version does, is what actually makes a structure immutable all the way down.

Reaching for an immutable structure

Reaching for an immutable structure
Need	Immutable choice
An ordered sequence that must not change	tuple instead of list
A set of members that must not change	frozenset instead of set
A small record with named fields	namedtuple, or a frozen dataclass
A dict key or set member (must be hashable)	tuple, frozenset — never list, dict, or set

Together

python

from collections import namedtuple
Point = namedtuple("Point", ["x", "y"])
p = Point(1, 2)
p.x = 5   # AttributeError — namedtuple fields cannot be reassigned

Remember: tuple, frozenset, namedtuple, and a frozen dataclass all refuse mutation — but a tuple holding a list is only immutable at the outer level, not inside.