Filter concepts by levelShowing all levels.

Python · Section 11

Standard Library

Level
intermediate
Read
165 min
Concepts
18

The modules a strong Python engineer knows well enough to avoid reinventing — from everyday containers and iteration tools to files, time, structured data, and concurrency.

What is true here

  1. collections, itertools, and functools replace common hand-written patterns with tested, fast building blocks.
  2. pathlib.Path replaces os.path's string functions with an object that joins, reads, and checks paths directly.
  3. logging is filterable and routable where print() is neither — reach for it in anything beyond a throwaway script.
  4. asyncio speeds up I/O-bound waiting; concurrent.futures.ProcessPoolExecutor is what actually parallelizes CPU-bound work.
  5. contextlib, dataclasses, abc, and typing each have a full section elsewhere — this section points to them rather than re-teaching them.

What you will be able to do

  • Reach for Counter, defaultdict, namedtuple, or deque instead of hand-rolling the same pattern with a plain dict or list
  • Chain, slice, and group iterators lazily with itertools instead of materializing intermediate lists
  • Cache expensive pure functions with @lru_cache and preserve a decorator's identity with @wraps
  • Build and inspect filesystem paths with pathlib.Path instead of os.path string functions
  • Attach a real IANA time zone with zoneinfo.ZoneInfo instead of a fixed UTC offset
  • Set up a named, leveled logger instead of debugging with print()
  • Write a minimal async def/await program and know when asyncio helps versus when it does not
  • Choose ThreadPoolExecutor or ProcessPoolExecutor correctly for I/O-bound versus CPU-bound work
  • Extract and replace text with re, and round-trip data through json
  • Define a fixed set of related values with enum.Enum instead of loose string constants
  • Parse command-line arguments declaratively with argparse instead of scanning sys.argv by hand
  • Know which narrow standard-library module (uuid, secrets, hashlib, tempfile, shutil, sqlite3, decimal, fractions, statistics) fits a given narrow problem

Data structures and iteration

Specialized containers, lazy iteration tools, and function utilities that replace common hand-written patterns.

collections

coreintermediate

collections supplies specialized containers — Counter, defaultdict, namedtuple, deque — that replace common hand-written patterns built from plain dicts and lists.

Think of it as

A plain dict or list is a blank container you configure by hand every time. collections is a drawer of pre-built containers, each one solving exactly one recurring problem: counting, auto-initializing, naming fields, or fast ends.

python
from collections import Counter, defaultdict, namedtuple, deque

Counter(iterable)
defaultdict(factory)
namedtuple("Name", ["field1", "field2"])
deque(iterable, maxlen=None)

What we're doing: Use Counter to tally word frequency and defaultdict to group results, the two most common collections patterns.

word_stats.pypython
from collections import Counter, defaultdict

words = "the quick fox the lazy fox the dog".split()

counts = Counter(words)
print(counts.most_common(2))

length_groups = defaultdict(list)
for word in words:
    length_groups[len(word)].append(word)
print(dict(length_groups))
5
Counter(words) tallies each string in one pass — no manual dict.get(word, 0) + 1 loop needed.
6
.most_common(2) returns the two highest counts as (item, count) tuples, already sorted.
9
length_groups[len(word)] never raises KeyError on a new length — defaultdict(list) creates [] automatically.
Output
[('the', 3), ('fox', 2)]
{3: ['the', 'the', 'the', 'fox', 'fox', 'dog'], 5: ['quick'], 4: ['lazy']}

Why this works: Counter and defaultdict both remove a manual "check if the key exists first" step. Counter treats every item as a key to tally; defaultdict runs its factory function exactly when a key is missing, so the append on line 10 always has a list to append to.

Using a plain dict and manually checking for a missing key

Wrong

python
groups = {}
for word in words:
    key = len(word)
    if key not in groups:
        groups[key] = []
    groups[key].append(word)

Better

python
from collections import defaultdict

groups = defaultdict(list)
for word in words:
    groups[len(word)].append(word)

What you see: Not a crash — working code, but four lines of boilerplate repeated at every call site that groups items into a dict of lists.

Why: defaultdict(list) moves the "is this key new?" check into the container itself, so every call site that groups data reads the same one line instead of reimplementing the check-then-initialize pattern.

Reach for the specialized container that matches the problem

Counting items

Counter

Grouping into a dict

defaultdict(list)

Named fields

namedtuple

Fast both ends

deque

  1. Counting items — Counter
  2. Grouping into a dict — defaultdict(list)
  3. Named fields — namedtuple
  4. Fast both ends — deque

collections — the containers worth knowing

collections — the containers worth knowing
NameWhat it is for
Countera dict subclass that counts hashable items; most_common(n) ranks them
defaultdict(factory)a dict that runs factory() to create a missing key instead of raising KeyError
namedtuple(name, fields)a tuple subclass with named fields — Point(x, y).x instead of p[0]
dequea double-ended queue — appendleft/popleft are O(1), unlike a list
OrderedDicta dict that remembers insertion order and adds move_to_end()
ChainMap(*maps)views several dicts as one, checked in order, without copying

Together

python
from collections import Counter, defaultdict, namedtuple

votes = Counter(["red", "blue", "red", "green", "red", "blue"])
print(votes.most_common(2))

by_dept = defaultdict(list)
for name, dept in [("Ada", "eng"), ("Sam", "eng"), ("Lee", "sales")]:
    by_dept[dept].append(name)
print(dict(by_dept))

Point = namedtuple("Point", ["x", "y"])
p = Point(3, 4)
print(p, p.x, p._asdict())

Remember: Counter tallies, defaultdict auto-creates missing values, namedtuple names fields, deque is fast at both ends — pick the one matching the shape of the problem.

See also: collections abc and typing · dictionaries · dataclass basics

itertools

coreintermediate

itertools provides fast, memory-efficient building blocks for looping — chaining several iterables, slicing lazily, grouping consecutive items, and generating combinations — without building an intermediate list.

Think of it as

A for loop processes one iterable at a time, fully, in memory if you are not careful. itertools is a set of lazy pipe fittings — chain joins pipes end to end, islice takes a slice without loading everything, groupby batches consecutive matching items, all without materializing the whole sequence first.

python
from itertools import chain, islice, groupby, product, combinations

chain(iter1, iter2, ...)
islice(iterable, stop)
groupby(iterable, key=None)

What we're doing: Combine two sensor logs into one stream, then group consecutive readings from the same device without loading every reading into a list first.

sensor_logs.pypython
from itertools import chain, groupby

log_a = [("dev1", 20), ("dev1", 21)]
log_b = [("dev2", 30), ("dev1", 19)]

combined = chain(log_a, log_b)
by_device = [(k, [v for _, v in g]) for k, g in groupby(combined, key=lambda r: r[0])]
print(by_device)
6
chain(log_a, log_b) walks log_a fully, then log_b — no list is built to hold both.
7
groupby only groups CONSECUTIVE matching keys — dev1 appears twice here but in two separate groups, because dev2 sits between them in the combined stream.
Output
[('dev1', [20, 21]), ('dev2', [30]), ('dev1', [19])]

Why this works: groupby scans left to right and starts a new group every time the key changes — it never looks ahead to merge non-adjacent matches. The second dev1 reading forms its own group because dev2 interrupted the run, which is the single most common groupby surprise.

Expecting groupby to group ALL matching items, not just consecutive ones

Wrong

python
from itertools import groupby

data = [("a", 1), ("b", 2), ("a", 3)]
result = [(k, list(v)) for k, v in groupby(data, key=lambda r: r[0])]
print(result)
# [('a', [1]), ('b', [2]), ('a', [3])] -- 'a' split into two groups!

Better

python
from itertools import groupby

data = [("a", 1), ("b", 2), ("a", 3)]
data.sort(key=lambda r: r[0])   # sort first so matches are adjacent
result = [(k, list(v)) for k, v in groupby(data, key=lambda r: r[0])]
print(result)
# [('a', [1, 3]), ('b', [2])]

What you see: The same key appears as two separate groups in the output, because the matching items were not next to each other in the input.

Why: groupby has no memory of keys it has already seen — it only compares each item to the one right before it. Sorting by the same key first guarantees every match is adjacent, which is what groupby actually requires to produce one group per key.

chain, then groupby — only CONSECUTIVE keys merge

chain(log_a, log_b)

dev1, dev1, dev2, dev1 — one lazy stream

groupby(combined, key)

starts a new group every time the key changes

3 groups, not 2

the second dev1 splits off — dev2 interrupted the run

  1. chain(log_a, log_b) — dev1, dev1, dev2, dev1 — one lazy stream
  2. groupby(combined, key) — starts a new group every time the key changes
  3. 3 groups, not 2 — the second dev1 splits off — dev2 interrupted the run

itertools — the functions worth knowing

itertools — the functions worth knowing
FunctionWhat it does
chain(*iterables)walks each iterable in turn as a single stream
islice(it, stop)lazily slices an iterator — works on generators, not just lists
groupby(it, key)groups consecutive items sharing the same key(item)
product(*iterables)the Cartesian product — every combination, order matters
combinations(it, r)every r-length selection, order does not matter, no repeats
count(start, step)an infinite arithmetic sequence — pair with islice or break

Together

python
from itertools import chain, islice, groupby

combined = list(chain([1, 2], [3, 4]))
print(combined)

first_three = list(islice(count_up_from_ten(), 3))
print(first_three)

rows = [("a", 1), ("a", 2), ("b", 3)]
grouped = [(k, list(v)) for k, v in groupby(rows, key=lambda r: r[0])]
print(grouped)

Remember: itertools chains and slices iterators lazily; groupby only groups items that are already adjacent — sort first if the matches are scattered.

See also: generator expressions · generator functions · functools module

functools

coreintermediate

functools provides tools for working with functions themselves — caching a function's results, pre-filling some of its arguments, and preserving its name and docstring through a decorator.

Think of it as

functools is a toolbox for modifying a function's behaviour without rewriting its body — lru_cache remembers past answers, partial pre-fills arguments, wraps repairs the identity a decorator would otherwise hide.

python
from functools import lru_cache, reduce, partial, wraps

@lru_cache(maxsize=128)
def expensive(n): ...

reduce(function, iterable, initial)
partial(function, *fixed_args)

What we're doing: Cache an expensive recursive Fibonacci function so repeated calls with the same argument skip recomputation entirely.

fib_cache.pypython
from functools import lru_cache

@lru_cache(maxsize=128)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(10))
print(fib.cache_info())
3
@lru_cache(maxsize=128) wraps fib so each distinct n is computed once and reused after that.
4
Without the cache, fib(10) makes 2*fib(9) recursive calls — an exponential blowup that the cache collapses to one call per unique n.
Output
55
CacheInfo(hits=8, misses=11, maxsize=128, currsize=11)

Why this works: Every recursive call to fib(n) for an n already computed returns instantly from the cache instead of recursing again. cache_info() shows 11 misses (one per unique n from 0 to 10) and 8 hits (repeated calls the recursion tree would otherwise redo).

Applying @lru_cache to a function with mutable or unhashable arguments

Wrong

python
from functools import lru_cache

@lru_cache(maxsize=128)
def total(items):        # items is a list
    return sum(items)

total([1, 2, 3])          # TypeError: unhashable type: 'list'

Better

python
from functools import lru_cache

@lru_cache(maxsize=128)
def total(items):        # items is a tuple instead
    return sum(items)

total((1, 2, 3))          # works — tuples are hashable

What you see: TypeError: unhashable type: 'list' raised the moment the cached function is called.

Why: lru_cache uses the arguments as a dict key internally, and dict keys must be hashable. Lists, dicts, and sets are all unhashable — pass an immutable equivalent (tuple, frozenset) instead.

functools — five tools, five distinct jobs

@lru_cache

remembers past return values by argument

reduce(fn, it, start)

folds a sequence into one value

partial(fn, *args)

pre-fills arguments, returns a new callable

@wraps(fn)

preserves __name__/__doc__ through a decorator

@singledispatch

dispatches by the type of the first argument

  • @lru_cache — remembers past return values by argument
  • reduce(fn, it, start) — folds a sequence into one value
  • partial(fn, *args) — pre-fills arguments, returns a new callable
  • @wraps(fn) — preserves __name__/__doc__ through a decorator
  • @singledispatch — dispatches by the type of the first argument

functools — the functions worth knowing

functools — the functions worth knowing
NameWhat it does
@lru_cache(maxsize=128)caches a function's return value per distinct argument set
@cacheunbounded shortcut for @lru_cache(maxsize=None), Python 3.9+
reduce(fn, iterable, start)cumulatively applies fn(acc, item), folding to one value
partial(fn, *args, **kwargs)returns a new callable with some arguments pre-filled
@wraps(fn)copies __name__/__doc__ onto a wrapper — essential inside any decorator
@singledispatchdispatches to a type-specific implementation, registered separately

Together

python
from functools import lru_cache, reduce, partial

@lru_cache(maxsize=128)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(10), fib.cache_info())
print(reduce(lambda acc, x: acc + x, [1, 2, 3, 4], 0))

add = lambda a, b: a + b
add_five = partial(add, 5)
print(add_five(10))

Remember: @lru_cache trades memory for speed on pure functions with hashable arguments; @wraps belongs on every decorator's inner function, without exception.

See also: function decorators · itertools module · higher order functions

collections.abc, abc, and typing

referenceintermediate

collections.abc defines the abstract base classes behind list/dict/set — Sequence, Mapping, Iterable — used to check what a type can do; abc and typing (declaring required methods, and type hints) each have their own full section elsewhere.

Think of it as

isinstance(x, list) checks one specific concrete type. isinstance(x, collections.abc.Sequence) checks a CAPABILITY — anything indexable and len()-able, including a custom class that is not a list at all. abc is the general machinery collections.abc's classes are built from; typing is a separate, static-analysis-time system for describing shapes to a type checker.

python
from collections.abc import Sequence, Mapping

isinstance(value, Sequence)

collections.abc — the abstract base classes worth knowing

collections.abc — the abstract base classes worth knowing
NameWhat it checks for
Iterablehas __iter__ — can be used in a for loop
Iteratorhas __iter__ and __next__ — an active, stateful iteration
Sequenceordered, indexable, and sized — like list or tuple
Mappingkey-based lookup, like dict — has __getitem__, keys(), etc.

Together

python
from collections.abc import Sequence

print(isinstance([1, 2, 3], Sequence))
print(isinstance("abc", Sequence))
print(isinstance({1, 2, 3}, Sequence))

Remember: collections.abc checks a CAPABILITY, broader than one concrete type — abc's enforcement mechanism and typing's hints each have their own full section.

See also: abc module · collections module

Advertisement

Files, time, and system interaction

Filesystem paths, dates and time zones, and talking to the operating system and other processes.

pathlib

corebeginner

pathlib represents a filesystem path as a Path object instead of a string — join pieces with /, read properties like .suffix and .parent, and call methods like .exists() directly on it.

Think of it as

os.path treats a path as a plain string you pass to standalone functions — os.path.join(a, b), os.path.splitext(p). pathlib treats a path as an object with its own properties and methods, so the same operations read as p / "b" and p.suffix.

python
from pathlib import Path

p = Path("dir") / "file.txt"
p.exists()
p.read_text()

What we're doing: Build a config file path, ensure its parent directory exists, and read the file if present, all without a single os.path string operation.

load_config.pypython
from pathlib import Path

config_path = Path("app") / "config" / "settings.toml"
config_path.parent.mkdir(parents=True, exist_ok=True)

if config_path.exists():
    print(config_path.read_text())
else:
    print(f"no config at {config_path}")
3
/ joins three segments into one Path — no os.path.join(...) call needed.
4
.parent is the directory containing the file; mkdir(parents=True) creates every missing level above it.
6
.exists() checks the real filesystem directly on the Path object.
Output
no config at app\config\settings.toml

Why this works: Path renders using the current OS's separator automatically — backslashes on Windows, forward slashes elsewhere — because the object stores segments, not a literal string, and only joins them for display or when handed to the OS.

Building paths with string concatenation instead of Path

Wrong

python
config_path = "app" + "/" + "config" + "/" + "settings.toml"
# breaks on Windows, where the separator is "\\" not "/"

Better

python
from pathlib import Path

config_path = Path("app") / "config" / "settings.toml"
# renders with the correct separator on every OS

What you see: Code that works on the author's machine fails to find files on a different OS, because the hardcoded separator does not match.

Why: Path stores path segments and only joins them with the correct OS separator when the path is turned into a string — string concatenation bakes in one specific separator permanently.

A Path's pieces

Path("data") / "reports" / "2026.csv"

Path("data")

root segment — the first component the path is built from

/ "reports"

join operator — / appends another path segment, OS-independent

"2026.csv"

final component — becomes .name; .stem is "2026", .suffix is ".csv"

  • Whole: Path("data") / "reports" / "2026.csv"
  • Path("data") — root segment: the first component the path is built from
  • / "reports" — join operator: / appends another path segment, OS-independent
  • "2026.csv" — final component: becomes .name; .stem is "2026", .suffix is ".csv"

pathlib.Path — the surface worth knowing

pathlib.Path — the surface worth knowing
MemberWhat it does
Path(*parts) / "next"builds a path; / joins another segment onto it
.name / .stem / .suffixthe final component / that component without its extension / just the extension
.parent / .partsthe containing directory / a tuple of every segment
.exists() / .is_file() / .is_dir()checks against the real filesystem
.read_text() / .write_text(s)reads or writes an entire text file in one call
.glob(pattern) / .iterdir()matching files by pattern / every entry in a directory
.mkdir(parents=True, exist_ok=True)creates a directory, including missing parents

Together

python
from pathlib import Path

p = Path("data") / "reports" / "2026.csv"
print(p.suffix, p.stem, p.parent)
print(p.with_suffix(".json"))

config_dir = Path.home() / ".config" / "myapp"
config_dir.mkdir(parents=True, exist_ok=True)

Remember: Path("a") / "b" joins segments correctly on every OS; use .suffix/.stem/.parent instead of slicing the string yourself.

See also: os sys subprocess · with statement · utility modules survey

datetime and zoneinfo

coreintermediate

datetime represents dates, times, and durations as objects instead of strings; zoneinfo attaches a real named time zone (like "America/New_York") that correctly accounts for daylight saving time.

Think of it as

A naive datetime (no time zone) is a wall clock with no label saying which city it is in — 14:30 alone is ambiguous. Attaching a zoneinfo.ZoneInfo turns it into a labeled clock: 14:30 in America/New_York is one specific, unambiguous instant.

python
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

dt = datetime(2026, 8, 21, 14, 30, tzinfo=ZoneInfo("Europe/London"))
dt + timedelta(hours=3)

What we're doing: Schedule a meeting in New York time and convert it to UTC for a log entry, using a real time zone rather than a fixed offset.

schedule.pypython
from datetime import datetime, timezone
from zoneinfo import ZoneInfo

meeting = datetime(2026, 8, 21, 9, 0, tzinfo=ZoneInfo("America/New_York"))
print(meeting.isoformat())

utc_time = meeting.astimezone(timezone.utc)
print(utc_time.isoformat())
4
tzinfo=ZoneInfo("America/New_York") attaches a real time zone, so this datetime knows it is EDT (UTC-4) in August.
7
.astimezone(timezone.utc) converts to UTC — the offset shift is computed from the zone's actual DST rules for this date.
Output
2026-08-21T09:00:00-04:00
2026-08-21T13:00:00+00:00

Why this works: ZoneInfo looks up the IANA time zone database to know that America/New_York is UTC-4 (EDT) in August but UTC-5 (EST) in January — a fixed-offset object could not make that distinction, and would produce the wrong UTC time on a date crossing a DST boundary.

Using a fixed UTC offset instead of a named zone for a recurring event

Wrong

python
from datetime import datetime, timedelta, timezone

# hardcoded -4, assuming EDT — wrong for a January meeting
ny_offset = timezone(timedelta(hours=-4))
meeting = datetime(2026, 1, 21, 9, 0, tzinfo=ny_offset)
# actually UTC-5 (EST) in January -- this is off by an hour

Better

python
from datetime import datetime
from zoneinfo import ZoneInfo

meeting = datetime(2026, 1, 21, 9, 0, tzinfo=ZoneInfo("America/New_York"))
# ZoneInfo applies the correct EST offset for a January date automatically

What you see: A meeting scheduled correctly in summer shows up an hour off once the calendar crosses a daylight-saving boundary.

Why: A fixed timezone(timedelta(...)) never changes, but real-world zones do twice a year. ZoneInfo consults the IANA database for the correct offset on the SPECIFIC date given, not a single offset baked in at creation time.

Fixed offset vs. ZoneInfo, same city

timezone(timedelta(hours=-4))

  • +Never changes — baked in at creation time
  • +Correct in August (EDT), wrong in January
  • +Off by an hour once DST flips

ZoneInfo("America/New_York")

  • −Consults the IANA database for the given date
  • −Applies EDT in summer, EST in winter, automatically
  • −Correct for any recurring or future-dated event
  • timezone(timedelta(hours=-4))
    • Never changes — baked in at creation time
    • Correct in August (EDT), wrong in January
    • Off by an hour once DST flips
  • ZoneInfo("America/New_York")
    • Consults the IANA database for the given date
    • Applies EDT in summer, EST in winter, automatically
    • Correct for any recurring or future-dated event

datetime and zoneinfo — the surface worth knowing

datetime and zoneinfo — the surface worth knowing
MemberWhat it does
date(year, month, day)a calendar date with no time component
datetime(y, m, d, h, mi, s, tzinfo=)a specific point in time, optionally time-zone aware
timedelta(days=, hours=, ...)a duration — add/subtract it from a date or datetime
.strftime(fmt) / .strptime(s, fmt)format a datetime to string / parse a string to datetime
.isoformat() / .fromisoformat(s)the standard "2026-08-21T14:30:00+00:00" form, both directions
ZoneInfo("America/New_York")a real IANA time zone, correctly applying DST rules for that date
timezone.utcthe fixed UTC offset — always available, no zone database needed

Together

python
from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

meeting = datetime(2026, 8, 21, 9, 0, tzinfo=ZoneInfo("America/New_York"))
print(meeting, meeting.tzname())

deadline = meeting + timedelta(days=7)
print(deadline.astimezone(timezone.utc))

Remember: ZoneInfo("Region/City") tracks real DST rules for that zone; a fixed-offset timezone does not — use ZoneInfo for anything recurring or future-dated.

See also: pathlib module · logging module

System interaction: os, sys, subprocess

standardintermediate

os reads environment variables and does OS-level file operations, sys exposes interpreter state like command-line arguments, and subprocess runs other programs and captures their output.

Think of it as

os is for talking to the operating system itself (environment, processes, low-level paths). sys is for talking to the Python interpreter running your code (its arguments, its module search path, how it exits). subprocess is for running an entirely different program and getting its output back.

python
import subprocess

result = subprocess.run(["ls", "-la"], capture_output=True, text=True)
print(result.stdout, result.returncode)

What we're doing: Run another Python process as a subprocess, capture its output as text, and check its exit code.

run_child.pypython
import sys
import subprocess

result = subprocess.run(
    [sys.executable, "-c", "print('hi from subprocess')"],
    capture_output=True,
    text=True,
)
print(result.stdout.strip())
print(result.returncode)
4
subprocess.run([...]) takes a list of arguments — the program name first, then each argument as a separate list item, not one shell string.
6
text=True decodes stdout/stderr as strings instead of raw bytes.
9
result.returncode is 0 because the child process exited successfully.
Output
hi from subprocess
0

Why this works: subprocess.run() blocks until the child process finishes, then returns a CompletedProcess object holding everything it produced — captured stdout/stderr as strings (because of text=True) and the process's exit code.

Passing a single shell string to subprocess.run() without shell=True

Wrong

python
import subprocess

# treated as ONE literal program name, not parsed into words
subprocess.run("ls -la /tmp")   # FileNotFoundError

Better

python
import subprocess

# a list — each argument is separate, no shell parsing needed
subprocess.run(["ls", "-la", "/tmp"])

# OR, only if a real shell feature (pipes, globbing) is required:
subprocess.run("ls -la /tmp", shell=True)

What you see: FileNotFoundError: no such file or directory — the whole string "ls -la /tmp" is treated as one program name to find.

Why: Without shell=True, subprocess.run expects a list where the first item is the program and the rest are its arguments, exactly as the OS process-creation API expects them — it does not parse a string into words the way a shell does.

os, sys, and subprocess — the surface worth knowing

os, sys, and subprocess — the surface worth knowing
MemberWhat it does
os.environ / os.environ.get(k, default)reads environment variables, with or without a KeyError on a missing key
os.getcwd() / os.path.join(a, b)the current working directory / joins path segments (pathlib.Path is preferred now)
sys.argvthe list of command-line arguments; index 0 is the script name
sys.exit(code)stops the process immediately with the given exit code
sys.paththe list of directories Python searches when importing a module
subprocess.run(cmd, capture_output=True, text=True)runs an external command, waits, and returns a CompletedProcess with .stdout/.returncode

Together

python
import os, sys, subprocess

print(os.environ.get("PATH") is not None)
print(sys.version_info[:2])

result = subprocess.run(
    [sys.executable, "-c", "print('hi from subprocess')"],
    capture_output=True,
    text=True,
)
print(result.stdout.strip(), result.returncode)

Remember: subprocess.run() takes a list of args, not a shell string, unless shell=True is deliberate; os.environ.get() avoids crashing on a missing variable.

See also: pathlib module · argparse module

contextlib

referenceintermediate

contextlib provides tools for building context managers — most commonly @contextmanager, which turns a generator function into a with-statement-compatible object without writing a full class.

Think of it as

Writing a context manager as a class means defining __enter__ and __exit__. contextlib.contextmanager is a shortcut: write one generator function, yield once where the with block's body should run, and contextlib builds the class-based protocol around it automatically.

python
from contextlib import contextmanager

@contextmanager
def resource():
    yield "value"

contextlib — the two most common tools

contextlib — the two most common tools
NameWhat it does
@contextmanagerwraps a generator function so it works in a with statement — code before yield is __enter__, after is __exit__
suppress(ExceptionType)a with block that silently ignores the given exception type if raised inside it

Together

python
from contextlib import contextmanager

@contextmanager
def timer(label):
    print(f"starting {label}")
    yield
    print(f"finished {label}")

with timer("upload"):
    print("uploading...")

Remember: @contextmanager turns one generator function into a with-compatible object — see Context Managers for the full class-based protocol it shortcuts.

See also: contextlib contextmanager · with statement

Advertisement

Concurrency and structured data

Running work concurrently, and the everyday formats — text patterns, JSON, fixed value sets — most programs handle directly.

asyncio

coreintermediate

asyncio runs many coroutines concurrently on a single thread — an async def function pauses at each await, letting other coroutines run while it waits on I/O instead of blocking the whole program.

Think of it as

A regular function call is one cook doing every step of one order start to finish. asyncio is one cook working several orders at once, switching to the next order every time the current one hits a waiting step — like water boiling — instead of standing idle.

python
import asyncio

async def main():
    result = await some_coroutine()
    return result

asyncio.run(main())

What we're doing: Fetch two resources concurrently with asyncio.gather instead of waiting for each one in sequence.

concurrent_fetch.pypython
import asyncio

async def fetch(name, delay):
    await asyncio.sleep(delay)
    return f"{name}-done"

async def main():
    results = await asyncio.gather(fetch("a", 0.01), fetch("b", 0.01))
    return results

print(asyncio.run(main()))
3
async def fetch(...) is a coroutine function — calling fetch("a", 0.01) does not run it yet, it returns a coroutine object.
4
await asyncio.sleep(delay) suspends THIS coroutine, letting the event loop run the other one during the wait.
8
gather runs both fetch coroutines concurrently — total wait time is close to the slower one, not the sum of both.
Output
['a-done', 'b-done']

Why this works: asyncio.run(main()) creates an event loop and runs main() to completion. Inside it, gather starts both fetch coroutines and lets each one's asyncio.sleep suspend without blocking the other — a real I/O wait (network, disk) behaves the same way sleep does here.

Calling a coroutine function without await and expecting it to run

Wrong

python
async def fetch(name):
    return f"{name}-done"

async def main():
    result = fetch("a")   # missing await
    print(result)          # prints a coroutine object, never runs

asyncio.run(main())

Better

python
async def fetch(name):
    return f"{name}-done"

async def main():
    result = await fetch("a")   # await actually runs it
    print(result)                 # "a-done"

asyncio.run(main())

What you see: Output is <coroutine object fetch at 0x...> and Python prints a RuntimeWarning: coroutine 'fetch' was never awaited.

Why: Calling an async def function only creates a coroutine object — it does not execute the function body. await is what actually drives the coroutine, exactly the way calling a generator function does not run it until something iterates it.

Waiting for two slow calls

Sequential (blocking)

  • +result_a = fetch(a) # waits fully
  • +result_b = fetch(b) # waits fully, after a
  • +Total time: time(a) + time(b)

asyncio.gather (concurrent)

  • −await asyncio.gather(fetch(a), fetch(b))
  • −Both coroutines run, switching at each await
  • −Total time: roughly max(time(a), time(b))
  • Sequential (blocking)
    • result_a = fetch(a) # waits fully
    • result_b = fetch(b) # waits fully, after a
    • Total time: time(a) + time(b)
  • asyncio.gather (concurrent)
    • await asyncio.gather(fetch(a), fetch(b))
    • Both coroutines run, switching at each await
    • Total time: roughly max(time(a), time(b))

asyncio — the surface worth knowing to start

asyncio — the surface worth knowing to start
NameWhat it does
async def f(): ...declares a coroutine function — calling it returns a coroutine object
await exprsuspends the current coroutine until expr completes, yielding control meanwhile
asyncio.run(coro())creates an event loop, runs one coroutine to completion, then closes the loop
asyncio.gather(*coros)runs several coroutines concurrently, returns all results once every one finishes
asyncio.sleep(seconds)a non-blocking wait — yields control instead of freezing the whole program
asyncio.create_task(coro())schedules a coroutine to run concurrently, without waiting for it immediately

Together

python
import asyncio

async def fetch(name, delay):
    await asyncio.sleep(delay)
    return f"{name}-done"

async def main():
    results = await asyncio.gather(fetch("a", 0.01), fetch("b", 0.01))
    return results

print(asyncio.run(main()))

Remember: await only yields control at an actual waiting point — asyncio speeds up I/O-bound waiting, never CPU-bound computation.

See also: concurrent futures module · generator functions · with statement

concurrent.futures

coreintermediate

concurrent.futures runs work across a pool of threads or processes behind one shared API — .submit() returns a Future you can .result() later, or .map() runs a function over many inputs at once.

Think of it as

ThreadPoolExecutor and ProcessPoolExecutor are two staffing choices behind the identical counter — same submit/map/result API either way. Threads share memory and suit I/O waiting; processes have separate memory and suit CPU-heavy work, because each process gets its own interpreter and its own GIL.

python
import concurrent.futures

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(fn, items))

What we're doing: Square four numbers across a thread pool using .map(), then submit a single call and read its result with .submit()/.result().

pool_demo.pypython
import concurrent.futures

def square(n):
    return n * n

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(square, [1, 2, 3, 4]))
print(results)

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
    future = pool.submit(square, 9)
    print(future.result())
7
.map(square, [1, 2, 3, 4]) schedules all four calls across the pool and returns results in input order.
11
.submit(square, 9) returns a Future immediately — the actual call may not have finished yet.
12
.result() blocks until that specific Future is done, then returns its value.
Output
[1, 4, 9, 16]
81

Why this works: The with block keeps the pool open until every submitted task completes, then shuts it down automatically on exit — this is what lets list(pool.map(...)) safely collect every result before the block ends.

Using ThreadPoolExecutor for CPU-bound work and expecting a speedup

Wrong

python
import concurrent.futures

def heavy_compute(n):
    return sum(i * i for i in range(n))  # pure CPU work

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(heavy_compute, [10**7] * 4))
    # barely faster than sequential -- the GIL serializes the CPU work

Better

python
import concurrent.futures

def heavy_compute(n):
    return sum(i * i for i in range(n))

with concurrent.futures.ProcessPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(heavy_compute, [10**7] * 4))
    # separate processes, separate GILs -- genuinely runs in parallel

What you see: Four "concurrent" CPU-heavy calls in a thread pool take roughly as long as running them one after another.

Why: Python threads still share one Global Interpreter Lock, which lets only one thread run Python bytecode at a time. Threads help when a task is waiting (I/O); for pure computation, only separate processes (each with its own interpreter and GIL) run in true parallel.

Choosing the pool

ThreadPoolExecutor

  • +Threads share one process's memory
  • +Best for I/O-bound: network calls, file/disk waits
  • +GIL still limits true CPU parallelism

ProcessPoolExecutor

  • −Each worker is a separate process, own memory
  • −Best for CPU-bound: heavy computation
  • −Bypasses the GIL — real parallel execution
  • ThreadPoolExecutor
    • Threads share one process's memory
    • Best for I/O-bound: network calls, file/disk waits
    • GIL still limits true CPU parallelism
  • ProcessPoolExecutor
    • Each worker is a separate process, own memory
    • Best for CPU-bound: heavy computation
    • Bypasses the GIL — real parallel execution

concurrent.futures — the surface worth knowing

concurrent.futures — the surface worth knowing
NameWhat it does
ThreadPoolExecutor(max_workers=)a pool of worker threads — best for I/O-bound tasks (network, disk)
ProcessPoolExecutor(max_workers=)a pool of worker processes — best for CPU-bound tasks, real parallelism
.submit(fn, *args)schedules one call, returns a Future immediately without blocking
.map(fn, iterable)schedules fn over every item, returns results in the same order as the input
future.result(timeout=None)blocks until the Future finishes, then returns its value (or re-raises its exception)
as_completed(futures)yields each Future as it finishes, not in submission order

Together

python
import concurrent.futures

def square(n):
    return n * n

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(square, [1, 2, 3, 4]))
print(results)

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
    future = pool.submit(square, 9)
    print(future.result())

Remember: ThreadPoolExecutor for I/O-bound waiting, ProcessPoolExecutor for CPU-bound work — submit everything before calling .result() on any of it.

See also: asyncio module · functools module

logging

coreintermediate

logging records leveled, structured messages — DEBUG through CRITICAL — that can be filtered, formatted, and routed to a file or console, unlike print() which always does exactly one thing.

Think of it as

print() is a megaphone with one setting: everything gets shouted, always, to the same place. logging is a mixing board — each message carries a severity level, and you decide per-handler what gets through and where it goes, without touching the code that generated the message.

python
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.info("processing order %s", order_id)

What we're doing: Set up a named logger with a formatter and confirm a WARNING-level message reaches the handler with the right shape.

inventory.pypython
import logging

logger = logging.getLogger("orders")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s:%(name)s:%(message)s"))
logger.addHandler(handler)

logger.warning("low stock for sku=%s", "A123")
4
setLevel(DEBUG) means this logger lets everything from DEBUG upward through — nothing is filtered out at this stage.
8
The %s placeholder and its argument are passed separately — logging only formats the string if the message actually gets emitted.
Output
WARNING:orders:low stock for sku=A123

Why this works: The Formatter's %(levelname)s:%(name)s:%(message)s pattern controls exactly how each field appears; %(message)s is the already-substituted "low stock for sku=A123" produced by combining the logger call's message and argument.

Building the message string before calling the logger

Wrong

python
logger.debug("processing " + str(order) + " with " + str(len(items)) + " items")
# the f-string/concatenation runs on EVERY call, even if DEBUG is filtered out

Better

python
logger.debug("processing %s with %d items", order, len(items))
# formatting only happens if this message actually gets emitted

What you see: No visible bug, but every debug call pays the cost of string building even when logging is configured to skip DEBUG entirely — measurable in a hot loop.

Why: Passing %s placeholders and arguments separately lets logging skip the formatting work entirely when the message's level is filtered out, since the string is only built at the point it is actually handed to a handler.

A message's trip through logging

logger.warning(msg, *args)

called in application code, checked against the logger's level

Logger

has a name and a level; decides whether the message proceeds at all

Handler

has its own level; decides WHERE the message goes (console, file, network)

Formatter

decides the final text shape before it is written

  1. logger.warning(msg, *args) — called in application code, checked against the logger's level
  2. Logger — has a name and a level; decides whether the message proceeds at all
  3. Handler — has its own level; decides WHERE the message goes (console, file, network)
  4. Formatter — decides the final text shape before it is written

logging — the core pieces

logging — the core pieces
NameWhat it does
getLogger(name)returns (or creates) a named logger; use __name__ per module
.debug/.info/.warning/.error/.critical(msg, *args)log at each severity level, ascending
.setLevel(level)the minimum severity this logger/handler lets through
StreamHandler / FileHandlersends formatted records to the console / a file
Formatter(fmt)controls the text shape — e.g. "%(levelname)s:%(name)s:%(message)s"
basicConfig(level=, format=)one-call setup for simple scripts — configures the root logger

Together

python
import logging

logger = logging.getLogger("orders")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s:%(name)s:%(message)s"))
logger.addHandler(handler)

logger.warning("low stock for sku=%s", "A123")

Remember: Pass %s placeholders and arguments separately to a logging call — the message is only built if it is actually emitted, unlike print() or an f-string.

See also: datetime and zoneinfo · try except · os sys subprocess

re (regular expressions)

coreintermediate

re matches text against a pattern — search() finds the first match anywhere, match() only at the start, findall() gets every match, and sub() replaces matches with new text.

Think of it as

A regular expression is a search template with wildcards — \d means "any digit," + means "one or more of the last thing." re compiles that template once and runs it against a string, instead of writing a hand-rolled character-by-character scanner.

python
import re

match = re.search(r"pattern", text)
if match:
    print(match.group())

What we're doing: Extract a phone number's area code and local part from free-form text using capture groups.

extract_phone.pypython
import re

text = "call 555-1234 now"
match = re.search(r"(\d{3})-(\d{4})", text)

if match:
    print(match.group(0))   # the whole match
    print(match.group(1))   # first group
    print(match.group(2))   # second group
4
r"(\d{3})-(\d{4})" is a raw string so \d is passed to re literally, not interpreted as a Python escape.
4
Each (...) captures one group — group(1) is the first parenthesized part, group(2) the second.
Output
555-1234
555
1234

Why this works: re.search scans the whole string for the first place the pattern matches. group(0) (or no argument) is always the entire match; group(1) and group(2) are the pieces captured by the two parenthesized groups, in order.

Forgetting the raw string prefix on a pattern with backslashes

Wrong

python
import re

# "\d" here is a Python escape sequence first, THEN a regex pattern —
# in this case it happens to still work, but "\b" or "\s" would not
pattern = "\d{3}-\d{4}"
print(re.search(pattern, "555-1234"))

Better

python
import re

# r"..." passes backslashes through unchanged to the regex engine
pattern = r"\d{3}-\d{4}"
print(re.search(pattern, "555-1234"))

What you see: Some escapes (like \b for a word boundary, or \A) silently mean something different or raise a DeprecationWarning/SyntaxWarning, because Python's own string escaping runs first.

Why: A non-raw string processes backslash escapes as Python string literals before re ever sees the pattern. r"..." disables that, so \d, \s, \b, and every other regex escape reaches re exactly as typed.

Reading a match, group by group

re.search(r"(\d{3})-(\d{4})", "call 555-1234 now")

re.search

searches anywhere — unlike re.match(), which anchors to index 0

(\d{3})

group(1) — first parenthesized capture — "555"

(\d{4})

group(2) — second parenthesized capture — "1234"

"call 555-1234 now"

subject text — the raw r"..." prefix passes backslashes through unchanged to this pattern

  • Whole: re.search(r"(\d{3})-(\d{4})", "call 555-1234 now")
  • re.search — searches anywhere: unlike re.match(), which anchors to index 0
  • (\d{3}) — group(1): first parenthesized capture — "555"
  • (\d{4}) — group(2): second parenthesized capture — "1234"
  • "call 555-1234 now" — subject text: the raw r"..." prefix passes backslashes through unchanged to this pattern

re — the functions and syntax worth knowing

re — the functions and syntax worth knowing
NameWhat it does
re.search(pattern, s)returns a Match for the first occurrence anywhere, or None
re.match(pattern, s)like search, but only matches at the very start of s
re.findall(pattern, s)returns every match as a list of strings (or tuples, with groups)
re.sub(pattern, repl, s)returns s with every match replaced by repl
re.compile(pattern)precompiles a pattern for reuse — faster when matched repeatedly
\d \w \s . * + ? {n,m}digit / word char / whitespace / any char / 0+ / 1+ / 0-1 / n-to-m repeats
(...) groups; m.group(n)captures part of the match, readable by position after a match

Together

python
import re

text = "call 555-1234 now"
m = re.search(r"(\d{3})-(\d{4})", text)
print(m.group(0), m.group(1), m.group(2))

emails = re.findall(r"\w+@\w+\.\w+", "a@x.com, b@y.org")
print(emails)

clean = re.sub(r"\s+", " ", "too   many    spaces")
print(clean)

Remember: Use a raw string r"..." for every pattern; re.search() looks anywhere, re.match() only at the start.

See also: json module · strings

json

standardbeginner

json.dumps() converts a Python dict/list into a JSON string; json.loads() parses a JSON string back into Python objects — the standard round trip for APIs and config files.

Think of it as

JSON is a text format both Python and almost every other language can read. dumps/loads are the two directions of a translator — Python objects going out as text, text coming back in as Python objects — with a small, fixed set of type mappings on each side.

python
import json

text = json.dumps(data, indent=2)
parsed = json.loads(text)

What we're doing: Serialize a user record to a JSON string, then parse it back and confirm the round trip preserves the data exactly.

json_roundtrip.pypython
import json

data = {"name": "Ada", "active": True, "tags": ["admin", "eng"], "score": 9.5}
text = json.dumps(data, indent=2, sort_keys=True)
print(text)

back = json.loads(text)
print(back == data)
4
sort_keys=True makes the key order deterministic — useful for diffing output or writing a stable test assertion.
7
json.loads parses the text back into a fresh dict — back == data compares by value, not identity.
Output
{
  "active": true,
  "name": "Ada",
  "score": 9.5,
  "tags": [
    "admin",
    "eng"
  ]
}
True

Why this works: json.dumps walks the dict and converts each Python type using json's fixed mapping — True becomes true, the list becomes a JSON array. json.loads reverses every step, so the parsed result is equal in value to the original, even though it is a different object.

Calling json.dumps on an object json does not know how to serialize

Wrong

python
import json
from datetime import date

data = {"created": date(2026, 8, 21)}
json.dumps(data)   # TypeError: Object of type date is not JSON serializable

Better

python
import json
from datetime import date

data = {"created": date(2026, 8, 21)}
json.dumps(data, default=lambda o: o.isoformat())   # converts date -> string first

What you see: TypeError: Object of type date is not JSON serializable, raised at the dumps() call.

Why: json only knows the built-in Python types listed in its mapping table — dict, list, str, int, float, bool, None. Anything else (date, a custom class, a set) needs a default= function telling dumps how to convert it to one of those first.

json — the functions and type mapping worth knowing

json — the functions and type mapping worth knowing
NameWhat it does
json.dumps(obj, indent=, sort_keys=)serializes a Python object to a JSON string
json.loads(s)parses a JSON string into Python objects
json.dump(obj, fp) / json.load(fp)same as dumps/loads, reading/writing a file object directly
dict ↔ objectJSON objects become Python dicts and back
list ↔ array, str ↔ stringstraightforward one-to-one mapping
True/False/None ↔ true/false/nullPython's booleans and None map to JSON's lowercase equivalents

Together

python
import json

data = {"name": "Ada", "active": True, "tags": ["admin", "eng"], "score": 9.5}
text = json.dumps(data, indent=2, sort_keys=True)
print(text)

parsed = json.loads(text)
print(parsed == data)

Remember: json.dumps/loads round-trip dicts, lists, strings, numbers, booleans, and None exactly — anything else needs a default= converter.

See also: re module · dataclass basics

enum

standardbeginner

enum.Enum defines a fixed, named set of related values as a real type — Status.PENDING is comparable, iterable, and unambiguous, unlike loose string constants like "pending".

Think of it as

A handful of string constants ("pending", "shipped") is a set of separate values Python has no way to relate to each other. Enum groups them into one type — Status.PENDING and Status.SHIPPED both belong to Status, so a typo like "pendign" fails immediately instead of silently comparing unequal forever.

python
from enum import Enum

class Status(Enum):
    PENDING = "pending"
    SHIPPED = "shipped"

What we're doing: Define an order status as an Enum, compare members, and look one up by its value.

order_status.pypython
from enum import Enum

class Status(Enum):
    PENDING = "pending"
    SHIPPED = "shipped"
    DELIVERED = "delivered"

def describe(status):
    if status == Status.PENDING:
        return "waiting to ship"
    return status.value

print(describe(Status.PENDING))
print(Status("shipped") is Status.SHIPPED)
3
class Status(Enum) groups three related constants into one type — Status.PENDING can never be confused with an unrelated string.
9
status == Status.PENDING compares by identity within the enum — safe, because Status.PENDING is the one and only PENDING member.
13
Status("shipped") looks up the member whose .value equals "shipped", returning Status.SHIPPED itself.
Output
waiting to ship
True

Why this works: Every Enum member is a singleton — there is exactly one Status.SHIPPED object, ever, for the lifetime of the program. That is why Status("shipped") is Status.SHIPPED is True: the lookup returns the same object, not a new equal one.

Comparing an Enum member to its raw value with ==

Wrong

python
from enum import Enum

class Status(Enum):
    PENDING = "pending"

status = Status.PENDING
if status == "pending":   # False! comparing Enum member to a plain string
    print("matched")
else:
    print("no match")

Better

python
from enum import Enum

class Status(Enum):
    PENDING = "pending"

status = Status.PENDING
if status == Status.PENDING:   # compare Enum to Enum
    print("matched")
if status.value == "pending":   # or compare the raw value explicitly
    print("matched via value")

What you see: status == "pending" is False even though Status.PENDING was defined with the value "pending" — silently skips the branch that should have matched.

Why: An Enum member is not equal to its raw value by default — Status.PENDING is a Status instance, not a str. Compare against the Enum member itself (Status.PENDING), or explicitly read .value first if a raw comparison is genuinely needed.

enum — the surface worth knowing

enum — the surface worth knowing
NameWhat it does
class X(Enum): NAME = valuedefines a fixed set of named members, each with an attached value
.name / .valuethe member's identifier string / its attached value
auto()assigns the next integer automatically — use when the value itself is arbitrary
list(X) / X.NAMEiterate every member / access one by its name
X(value)looks up the member whose .value matches, e.g. Status("pending")
IntFlaga variant whose members combine with | into bitmask-style flag sets

Together

python
from enum import Enum, auto

class Status(Enum):
    PENDING = "pending"
    SHIPPED = "shipped"
    DELIVERED = "delivered"

print(Status.PENDING, Status.PENDING.value)
print(list(Status))
print(Status("shipped") is Status.SHIPPED)

class Color(Enum):
    RED = auto()
    GREEN = auto()
print(Color.RED.value, Color.GREEN.value)

Remember: Compare Enum members to each other (Status.PENDING), not to their raw .value — the member is the identity, the value is just its payload.

See also: json module · dataclass basics

Advertisement

CLI parsing and utility modules

Building a command-line interface, exact numeric types, and a handful of narrow single-purpose modules worth knowing exist.

argparse

standardintermediate

argparse builds a command-line interface declaratively — add_argument() describes each flag or positional value once, and argparse handles parsing, type conversion, defaults, and --help text.

Think of it as

Parsing sys.argv by hand means writing an if/elif chain checking for each flag string, converting types manually, and writing help text separately. argparse is a form builder — describe each field once (name, type, default), and the parser, the validation, and the help text all come from that single description.

python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("input")
parser.add_argument("--verbose", action="store_true")
args = parser.parse_args()

What we're doing: Build a small CLI with one required positional argument and two optional flags, one boolean and one typed.

uploader.pypython
import argparse

parser = argparse.ArgumentParser(prog="uploader")
parser.add_argument("filename")
parser.add_argument("--verbose", action="store_true")
parser.add_argument("--retries", type=int, default=3)

args = parser.parse_args(["report.csv", "--verbose", "--retries", "5"])
print(args.filename, args.verbose, args.retries)
4
"filename" with no dashes is positional — required, matched by position, not by a flag name.
5
action="store_true" makes --verbose a switch: True if present on the command line, False (the implicit default) if absent.
6
type=int converts the string "5" to the integer 5 automatically — args.retries is already an int, not a string.
Output
report.csv True 5

Why this works: parse_args() matches "report.csv" to the positional filename argument, recognizes --verbose as present (setting it True), and converts the string after --retries using the type=int declared on that argument — all from the single add_argument() call per field.

Forgetting default= on an optional argument and getting None instead of a sensible fallback

Wrong

python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--retries", type=int)   # no default

args = parser.parse_args([])
print(args.retries + 1)   # TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Better

python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--retries", type=int, default=3)

args = parser.parse_args([])
print(args.retries + 1)   # 4 -- default fills in when the flag is omitted

What you see: TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' the moment the omitted flag's value is used.

Why: An optional argument with no default= is None when the flag is not passed — argparse never invents a default on its own. Any optional argument the code later uses arithmetically or as a string needs an explicit default=.

argparse — the surface worth knowing

argparse — the surface worth knowing
MemberWhat it does
ArgumentParser(prog=, description=)creates the parser; prog/description feed into the generated --help text
add_argument("name")a required positional argument, read by position
add_argument("--flag")an optional named argument, e.g. --retries 5
action="store_true"makes a flag a boolean switch — present means True, absent means False
type=int / default=valueconverts the raw string automatically / supplies a value when the flag is omitted
parse_args() / parse_args([...])parses sys.argv[1:] by default, or an explicit list of strings

Together

python
import argparse

parser = argparse.ArgumentParser(prog="uploader")
parser.add_argument("filename")
parser.add_argument("--verbose", action="store_true")
parser.add_argument("--retries", type=int, default=3)

args = parser.parse_args(["report.csv", "--verbose", "--retries", "5"])
print(args.filename, args.verbose, args.retries)

Remember: Declare each argument once with add_argument(); argparse handles parsing, defaults, and --help — a hand-rolled sys.argv scan reimplements all of it, worse.

See also: os sys subprocess

dataclasses

referenceintermediate

dataclasses generates __init__, __repr__, and __eq__ automatically from a class's type-annotated fields, replacing the boilerplate of writing a plain data-holding class by hand.

Think of it as

A plain class holding a few fields needs __init__ to assign them, __repr__ to print them usefully, and __eq__ to compare them — three methods that just restate the field list. @dataclass reads that field list once (from the annotations) and generates all three.

python
from dataclasses import dataclass

@dataclass
class Point:
    x: int
    y: int

dataclasses — the one decorator worth knowing here

dataclasses — the one decorator worth knowing here
NameWhat it does
@dataclassgenerates __init__, __repr__, __eq__ from a class's annotated fields
@dataclass(frozen=True)the same, plus makes every field immutable after construction

Together

python
from dataclasses import dataclass

@dataclass
class Point:
    x: int
    y: int

p1 = Point(3, 4)
p2 = Point(3, 4)
print(p1, p1 == p2)

Remember: @dataclass generates __init__/__repr__/__eq__ from annotated fields — see Dataclasses and Data Modeling for slots, inheritance, and frozen instances.

See also: dataclass basics · enum module

Numeric types beyond float: decimal, fractions, statistics

standardintermediate

decimal gives exact base-10 arithmetic for money, fractions gives exact rational numbers, and statistics computes mean/median/standard deviation without a third-party library.

Think of it as

A float stores numbers in binary, so 0.1 + 0.2 is not exactly 0.3 — a rounding error invisible until it compounds. decimal and fractions are both exact: decimal stores base-10 digits directly (what money needs), fractions stores an exact numerator/denominator pair (what a ratio needs). statistics is a third, unrelated convenience: common summary numbers over a list, already in the standard library.

python
from decimal import Decimal

total = Decimal("19.99") + Decimal("5.01")   # exact, always from a string

What we're doing: Add two money amounts with Decimal and show the same addition drifting under plain float arithmetic.

money_math.pypython
from decimal import Decimal

exact = Decimal("0.1") + Decimal("0.2")
print(exact, exact == Decimal("0.3"))

inexact = 0.1 + 0.2
print(inexact)
3
Decimal("0.1") is constructed from a string, so it stores the exact base-10 value 0.1 — not float's nearest binary approximation.
7
0.1 + 0.2 as plain floats does not equal 0.3 exactly — a classic binary floating-point rounding artifact.
Output
0.3 True
0.30000000000000004

Why this works: Decimal stores digits in base 10 internally, matching how humans write money, so 0.1 and 0.2 are exact values with an exact sum. float stores numbers in base 2, where 0.1 has no exact binary representation, so the tiny rounding error becomes visible once printed at full precision.

Constructing a Decimal from a float instead of a string

Wrong

python
from decimal import Decimal

price = Decimal(0.1)   # float 0.1's imprecision is now baked in
print(price)            # 0.1000000000000000055511151231257827021181583404541015625

Better

python
from decimal import Decimal

price = Decimal("0.1")   # exact -- constructed from the string directly
print(price)               # 0.1

What you see: Decimal(0.1) prints as a long, ugly, imprecise number instead of the exact 0.1 that was intended.

Why: Decimal(float) converts the float's ALREADY-imprecise binary value into decimal digits, faithfully preserving the error rather than fixing it. Decimal("0.1") skips floats entirely and parses the exact decimal string.

decimal, fractions, and statistics — the surface worth knowing

decimal, fractions, and statistics — the surface worth knowing
Module.memberWhat it does
decimal.Decimal("0.1")an exact base-10 number — construct from a string to avoid float imprecision
decimal.Decimal + Decimalexact arithmetic — no binary rounding error, unlike float + float
fractions.Fraction(1, 3)an exact rational number stored as numerator/denominator
fractions.Fraction + Fractionexact arithmetic that stays a precise fraction, never rounds
statistics.mean(data) / .median(data)the average / the middle value of a list of numbers
statistics.stdev(data)the sample standard deviation of a list of numbers

Together

python
from decimal import Decimal
from fractions import Fraction
import statistics

price = Decimal("19.99") + Decimal("5.01")
print(price)

part = Fraction(1, 3) + Fraction(1, 6)
print(part)

scores = [2, 4, 4, 4, 5, 5, 7, 9]
print(statistics.mean(scores), statistics.median(scores))

Remember: Construct Decimal from a string, never a float, for exact arithmetic — use it for money, Fraction for exact ratios, plain float for everything else.

See also: numbers

Utility modules: sqlite3, uuid, secrets, tempfile, shutil, hashlib

referenceintermediate

Six standard-library modules, each solving one narrow problem: an embedded database (sqlite3), unique ids (uuid), secure randomness (secrets), scratch files (tempfile), file copying (shutil), and hashing (hashlib).

Think of it as

Each of these is a single-purpose tool reached for occasionally, not built up into a mental model the way collections or itertools are — know that each one exists and what problem it solves, then look up the exact call when the need comes up.

python
import uuid, hashlib

user_id = uuid.uuid4()
digest = hashlib.sha256(b"data").hexdigest()

What we're doing: Generate a unique id and a secure token, hash a value, and copy a file through a temporary directory — one call from each module.

utility_tour.pypython
import uuid, secrets, hashlib, tempfile, shutil, os

user_id = uuid.uuid4()
token = secrets.token_hex(8)
digest = hashlib.sha256(b"hello world").hexdigest()

with tempfile.TemporaryDirectory() as d:
    src = os.path.join(d, "a.txt")
    open(src, "w").write("data")
    dst = os.path.join(d, "b.txt")
    shutil.copy(src, dst)
    print(user_id.version, len(token), digest[:8], os.path.exists(dst))
3
uuid.uuid4() generates a random 128-bit id — user_id.version confirms it is version 4 (random-based).
4
secrets.token_hex(8) returns 16 hex characters (8 bytes) suitable for a session token, using a cryptographically secure source.
7
TemporaryDirectory() as a context manager deletes the directory and everything in it automatically when the block exits.
Output
4 16 b94d27b9 True

Why this works: Each call does exactly one narrow job: uuid4() and token_hex() both produce randomness, but only secrets is safe for anything security-sensitive; hashlib.sha256().hexdigest() always returns a fixed 64-character hex string regardless of input size; shutil.copy() and the TemporaryDirectory context manager together avoid leaving scratch files behind.

Using the random module instead of secrets for a token or password

Wrong

python
import random

# random is predictable given its internal state -- NOT secure
token = "".join(random.choices("0123456789abcdef", k=16))

Better

python
import secrets

# secrets uses the OS's cryptographically secure random source
token = secrets.token_hex(8)

What you see: No visible error — the code runs fine, but the token is predictable to an attacker who can observe enough of random's output or knows its seed.

Why: random is a deterministic pseudorandom generator meant for simulations and games, not security — its internal state can potentially be inferred from enough output. secrets is built on the OS's cryptographically secure random source specifically for tokens, passwords, and similar values.

Six utility modules, one row each

Six utility modules, one row each
ModuleWhat it is forCore call
sqlite3an embedded SQL database — no separate server processsqlite3.connect(path).cursor()
uuidgenerates a unique identifier, effectively collision-freeuuid.uuid4()
secretscryptographically secure randomness — for tokens, not gamessecrets.token_hex(16)
tempfilescratch files/directories that clean themselves uptempfile.TemporaryDirectory()
shutilhigh-level file operations: copy, move, disk usageshutil.copy(src, dst)
hashlibcryptographic hash digests of byteshashlib.sha256(data).hexdigest()

Together

python
import sqlite3, uuid, secrets, tempfile, shutil, hashlib, os

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
conn.execute("INSERT INTO users (name) VALUES (?)", ("Ada",))
print(conn.execute("SELECT * FROM users").fetchall())

print(uuid.uuid4())
print(secrets.token_hex(8))

with tempfile.TemporaryDirectory() as d:
    path = os.path.join(d, "a.txt")
    open(path, "w").write("data")
    shutil.copy(path, os.path.join(d, "b.txt"))
    print(os.path.exists(os.path.join(d, "b.txt")))

print(hashlib.sha256(b"hello world").hexdigest())

Remember: secrets (not random) for anything security-sensitive; tempfile.TemporaryDirectory() as a context manager for guaranteed cleanup.

See also: pathlib module · with statement

Advertisement