Python

A readable, batteries-included language for backends, data work and automation.

Python is a general-purpose programming language designed to be read easily. You write instructions in near-plain English, and an interpreter runs them straight away — there is no separate build step to sit through.

Difficulty: beginner
Time: 40+ hours
Sections written: 39

Why it matters

It is the default language for data work, scripting and automation, so it is often the fastest way to get something working.
The standard library covers files, dates, JSON, HTTP and testing without installing anything.
Readable code stays cheap to change, which matters far more over a project’s life than how fast it was to write.
Django and FastAPI are both Python, so the language carries straight into two of the other topics here.

Where it is used

Web backends — Django, FastAPI, Flask
Data analysis and machine learning — pandas, NumPy, PyTorch
Automation and glue scripts, from build steps to one-off data fixes
Infrastructure tooling — Ansible, AWS CDK, most cloud SDKs

The big picture

How the pieces of the language fit together

Names

labels bound to objects

Objects

everything is one

Control flow

if, for, while

Functions

reusable behaviour

Modules

files you import

Sections

Common patterns

Context manager for guaranteed cleanup

intermediate

python

from contextlib import contextmanager
import time

@contextmanager
def timed_operation(name):
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        print(f"{name} took {elapsed:.3f}s")

with timed_operation("load_config"):
    load_config()

Use when

You acquire a resource — a file, a lock, a database connection — that must be released even if the code in between raises.

Do not use when

The resource has no cleanup step, or is a short-lived local value with no external handle to release.

`@contextmanager` turns a generator into a context manager: everything before `yield` runs on entry, everything after runs on exit — including when an exception passes through, because `finally` always runs. That is less code than a class with `__enter__` and `__exit__` for a case this simple.

Retry with backoff, as a decorator

intermediate

python

import time
import functools

def retry(attempts=3, base_delay=0.5):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return func(*args, **kwargs)
                except ConnectionError:
                    if attempt == attempts:
                        raise
                    time.sleep(base_delay * 2 ** (attempt - 1))
        return wrapper
    return decorator

@retry(attempts=3)
def fetch_status(order_id):
    return http_get(f"/orders/{order_id}/status")

Use when

A call to an external service — an API, a database — can fail transiently, and succeeding on a second or third attempt is normal.

Do not use when

The failure is deterministic, like bad input or wrong credentials. Retrying a call that cannot succeed only delays the real error and can add load to a struggling service.

`functools.wraps` keeps the wrapped function's name and docstring, which matters for debugging. The delay doubles each attempt — 0.5s, 1s, 2s — so a struggling service sees falling pressure instead of an immediate second hit. Catching `ConnectionError` specifically, not a bare `except:`, means a bug in your own code still raises immediately.

Dataclass for a value object

beginner

python

from dataclasses import dataclass

@dataclass(frozen=True)
class Money:
    amount: int  # smallest currency unit, e.g. cents
    currency: str

    def __add__(self, other):
        if self.currency != other.currency:
            raise ValueError("cannot add different currencies")
        return Money(self.amount + other.amount, self.currency)

price = Money(1999, "USD")
tax = Money(160, "USD")
total = price + tax

Use when

A class exists mainly to hold a fixed set of related fields, with little behavior beyond equality and a readable repr.

Do not use when

`@dataclass` would fight with custom validation on every field assignment, or with a base class that already defines its own `__init__`.

`@dataclass` generates `__init__`, `__repr__`, and `__eq__` from the field list, so two `Money` instances with the same amount and currency compare equal without writing that logic by hand. `frozen=True` makes instances immutable, matching how a real amount behaves — you create a new value, you do not mutate an old one.

Generator pipeline for streaming data

intermediate

python

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def parse_json_lines(lines):
    import json
    for line in lines:
        if line.strip():
            yield json.loads(line)

def filter_errors(records):
    for record in records:
        if record.get("level") == "ERROR":
            yield record

errors = filter_errors(parse_json_lines(read_lines("app.log")))
for error in errors:
    print(error["message"])

Use when

You process a sequence too large to hold in memory at once — a large file, a paginated API — and each stage only needs to see one item at a time.

Do not use when

The data is small, or you need to access it more than once or out of order. A generator is single-pass and cannot be rewound or indexed.

Each function is a generator, so calling it does no work — it just returns an iterator. Work happens lazily, one line at a time, as the final `for` loop pulls values through the whole chain. Only one line is ever in memory, regardless of file size.

Dependency injection through arguments

beginner

python

from datetime import datetime, timezone

def is_expired(expires_at, now=None):
    now = now or datetime.now(timezone.utc)
    return now >= expires_at

# Production: uses the real clock.
is_expired(token.expires_at)

# Test: pins the clock, no monkeypatching needed.
is_expired(token.expires_at, now=datetime(2026, 1, 1, tzinfo=timezone.utc))

Use when

A function calls something hard to test directly — the current time, a database, an HTTP client — and you want to substitute a fake in tests without a mocking framework.

Do not use when

The dependency is a pure, deterministic function with no external state. Injecting something like `str.upper` adds a parameter for no testing benefit.

The dependency — here, the current time — arrives as a parameter with a sensible default, rather than being looked up inside the function body. A test can pass a fixed value and get a deterministic result, with no need to patch `datetime.now` globally.

Common mistakes

Mutable default argument

Wrong

python

def add_item(item, cart=[]):
    cart.append(item)
    return cart

Better

python

def add_item(item, cart=None):
    if cart is None:
        cart = []
    cart.append(item)
    return cart

What you see: Calling `add_item('apple')` twice in a row returns `['apple', 'apple']` from the second call, even though a fresh empty cart was expected each time.

Why: A default argument is evaluated once, when the function is defined — not on every call. Every call that omits `cart` shares and mutates that same list, so items from one call leak into the next.

Catching a bare `except:`

Wrong

python

try:
    process_order(order)
except:
    log.error("order failed")

Better

python

try:
    process_order(order)
except (ValueError, ConnectionError) as e:
    log.error("order failed: %s", e)

What you see: The program becomes impossible to interrupt from the terminal, and a real bug's traceback never appears — only the generic "order failed" message.

Why: A bare `except:` catches everything, including `KeyboardInterrupt` and `SystemExit`, so Ctrl-C stops working — and a typo like a misspelled variable name (`NameError`) gets silently swallowed instead of surfacing as a bug.

Modifying a list while iterating over it

Wrong

python

for item in items:
    if item.is_expired():
        items.remove(item)

Better

python

items[:] = [item for item in items if not item.is_expired()]

What you see: Roughly half of the expired items remain after the loop finishes, and the exact count depends on how many consecutive expired items sit next to each other.

Why: Removing an element shifts every later index down by one, but the loop's internal position keeps advancing — so the element right after a removed one gets skipped entirely.

Comparing floats with `==`

Wrong

python

total = 0.1 + 0.2
if total == 0.3:
    print("matched")

Better

python

import math

if math.isclose(total, 0.3):
    print("matched")

What you see: `0.1 + 0.2 == 0.3` evaluates to `False`, and a total that prints as correct still fails an exact equality check.

Why: Most decimal fractions have no exact binary floating-point representation, so `0.1 + 0.2` evaluates to `0.30000000000000004`, not `0.3`. `math.isclose` compares within a tolerance instead of demanding bit-for-bit equality.

Not closing a file or connection explicitly

Wrong

python

f = open("report.csv", "w")
f.write(row)

Better

python

with open("report.csv", "w") as f:
    f.write(row)

What you see: The file exists but is empty or truncated, especially when the script exits right after writing, or a later process cannot open the file because a handle is still held.

Why: Without `with`, the file stays open until the object is garbage collected — which is not guaranteed to happen immediately, especially after an exception skips the close. Buffered writes may never reach disk if the process exits first.

Best practices

Use `pathlib.Path` instead of string concatenation for file paths.
`Path` objects handle platform separators correctly and chain naturally — `path / 'subdir' / 'file.txt'` works on both Windows and Linux, where `'dir' + '/' + 'file'` breaks on Windows.
Write f-strings for string formatting instead of `%` or `.format()`.
f-strings evaluate expressions inline (`f'{user.name}'`), are the fastest of the three formatting styles, and are what nearly all Python written since 3.6 uses.
Name booleans and boolean-returning functions as predicates: `is_valid`, `has_permission`.
A reader can tell from the name alone that `if is_valid:` reads as a yes/no question, without opening the function.

Avoid

Avoid `import *`.
It pulls every public name from a module into your namespace, so a reader — and your editor's autocomplete — cannot tell where a name like `resolve` came from, and two modules can silently overwrite each other's exports.
Avoid deep inheritance hierarchies of more than two or three levels.
Each extra level makes it harder to know which class defines the method actually running, especially once `super()` calls are mixed in — composition scales more predictably than a tall inheritance chain.
Avoid catching exceptions you cannot handle meaningfully.
A caught exception that just gets re-raised or logged and ignored hides the failure from whoever calls your function, and from monitoring that expects to see it.

Performance

Profile before optimizing — use `cProfile` or `timeit`, not intuition.
The slow part of a program is often not where it looks slow; a five-minute profiling run regularly finds the real bottleneck in a different function than the one you assumed.
Prefer a generator over a list when you only iterate once.
A generator holds one item in memory at a time; building the full list first costs memory proportional to its size for no benefit if you were only going to loop over it once.
Batch database and network calls instead of calling once per item in a loop.
One query for 100 rows is one round trip; a query per row inside a loop is 100 round trips, and network latency usually costs more than the query itself.

Maintainability

Keep functions short enough to see on one screen.
A function you cannot see all of forces you to hold its earlier lines in memory while reading the rest, which is where bugs at the boundaries between sections tend to hide.
Add type hints to function signatures in shared code.
A hint like `def parse(raw: str) -> dict` tells a caller what to pass and what comes back without opening the function body, and lets `mypy` catch a mismatched call before it runs.
Write the docstring before the function body, not after.
Writing what the function should do first catches a function trying to do two unrelated things — if the docstring needs "and", the function probably does too much.

Security

Never build a SQL query with string formatting or concatenation.
`f"SELECT * FROM users WHERE id = {user_id}"` lets an attacker who controls `user_id` inject arbitrary SQL. A parameterized query — `cursor.execute(sql, (user_id,))` — sends the value separately from the query structure.
Load secrets from environment variables or a secrets manager, never hardcode them.
A hardcoded API key or password ends up in version control history permanently, even after you delete it in a later commit — it survives in every prior commit.
Validate any input that reaches `subprocess`, `eval`, or `pickle.loads`.
All three execute what you hand them: `subprocess` with `shell=True` can run arbitrary shell commands, `eval` runs arbitrary Python, and unpickling untrusted data can execute arbitrary code during deserialization.

Testing

Test behavior through the public interface, not internal implementation details.
A test that reaches into a private attribute breaks the moment you rename that attribute, even when the function's actual behavior — its inputs and outputs — never changed.
Use `pytest` fixtures instead of repeating setup code in every test.
A fixture defines setup once and pytest injects it into every test that asks for it, so a change to how a test database is created happens in one place instead of in every test file.
Assert on the specific value you expect, not just that no exception was raised.
A test that only checks the result is not `None` passes even if the function returns completely wrong data, as long as it returns something.

Real-world scenarios

A resilient client for a flaky payment provider

advanced

Problem

Your checkout flow calls a third-party payment API that times out under load roughly 2% of the time, and a raw timeout currently surfaces as a failed order even though the charge often went through on the provider's side.

Requirements

Retry only errors that are safe to retry — never retry after the provider may already have charged the card.
Never charge a customer twice for the same order.
Fail fast and clearly when the provider is down for an extended period, rather than making every request wait out its own timeout.

Approach

Generate an idempotency key from the order ID before the first attempt. — The payment provider deduplicates requests carrying the same idempotency key, so a retried request after a timeout cannot create a second charge even if the first attempt actually succeeded.
Wrap the call in a decorator that retries only on connection and timeout errors, with exponential backoff. — A 4xx response — a bad card number, insufficient funds — is deterministic. Retrying it wastes time and produces the identical failure again.
Add a circuit breaker that stops calling the provider for 30 seconds after 5 consecutive failures. — If the provider is genuinely down, retrying from every checkout adds load to a service that is already struggling, and makes every customer wait through a doomed retry loop instead of failing quickly.

Result

With the idempotency key in place, a retried request after a timeout returns the original charge's result instead of creating a second one — the customer is charged once no matter how many attempts the retry logic makes. Once the breaker opens, checkout fails fast with a clear "payment provider unavailable" error instead of every concurrent request hanging through its own three-attempt retry, which is what keeps the rest of the site responsive during a provider outage.

Implementation

Idempotent charge with retry

python

import time
import functools

def retry_transient(attempts=3, base_delay=0.5):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return func(*args, **kwargs)
                except (ConnectionError, TimeoutError):
                    if attempt == attempts:
                        raise
                    time.sleep(base_delay * 2 ** (attempt - 1))
        return wrapper
    return decorator

@retry_transient(attempts=3)
def charge_card(order_id, amount_cents, card_token):
    idempotency_key = f"order-{order_id}"
    return payment_api.charge(
        amount=amount_cents,
        card=card_token,
        idempotency_key=idempotency_key,
    )

The idempotency key is derived from `order_id`, so every retry for the same order sends the identical key. The provider recognizes a repeated key and returns the original charge's result instead of creating a new one — that is what makes the retry safe.

Circuit breaker around the client

python

import time

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_after=30):
        self.failure_threshold = failure_threshold
        self.reset_after = reset_after
        self.failures = 0
        self.opened_at = None

    def call(self, func, *args, **kwargs):
        if self.opened_at and time.monotonic() - self.opened_at < self.reset_after:
            raise CircuitOpen("payment provider unavailable, try again shortly")
        try:
            result = func(*args, **kwargs)
        except (ConnectionError, TimeoutError):
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.opened_at = time.monotonic()
            raise
        else:
            self.failures = 0
            self.opened_at = None
            return result

breaker = CircuitBreaker()
breaker.call(charge_card, order.id, order.total_cents, order.card_token)

The breaker tracks consecutive failures, not a rolling average — five in a row opens it, and one success resets the count to zero. While open, `call` fails in microseconds instead of waiting out another timeout, which protects the rest of checkout from a single bad dependency.

Generating a report from 2 million rows without exhausting memory

advanced

Problem

A nightly export writes every order to a CSV file, but loading all rows into a list before writing crashed the worker with an out-of-memory error once the table passed a few hundred thousand rows.

Requirements

Process the full table without holding more than a small, constant amount of it in memory at once.
Resume from where it stopped if the worker is interrupted partway through.
Produce output rows in the same format regardless of table size.

Approach

Fetch rows from the database in fixed-size pages instead of one query for the whole table. — A single unpaginated query either loads the entire result set into memory at once or keeps a server-side cursor open for the whole run, depending on the driver — both scale badly past a few hundred thousand rows.
Turn the page fetcher into a generator that yields one row at a time to the CSV writer. — The writer never needs to know how many rows exist in total or hold them all at once — it just consumes whatever the generator produces next.
Record the last processed ID after every page and read it back on startup. — Restarting from row zero after a crash would duplicate a report the export already half-produced; resuming from the last checkpoint does not.

Result

Memory use during the export stays flat regardless of table size, since only one page of rows is ever held at once. If the worker is killed and restarted, it resumes within 5,000 rows of where it stopped instead of re-running the whole export from the beginning.

Implementation

Paginated generator over the table

python

def fetch_orders_since(conn, last_id, page_size=1000):
    while True:
        rows = conn.execute(
            "SELECT id, user_id, total_cents, created_at FROM orders "
            "WHERE id > %s ORDER BY id LIMIT %s",
            (last_id, page_size),
        ).fetchall()
        if not rows:
            return
        for row in rows:
            yield row
        last_id = rows[-1]["id"]

Each page is a small, bounded query — never more than `page_size` rows in memory. The generator yields individual rows, so from the caller's side it looks like iterating a single, arbitrarily long sequence, with the pagination invisible.

Resumable export loop

python

import csv
import json

def run_export(conn, checkpoint_path, output_path):
    last_id = load_checkpoint(checkpoint_path)
    with open(output_path, "a", newline="") as f:
        writer = csv.writer(f)
        for row in fetch_orders_since(conn, last_id):
            writer.writerow([row["id"], row["user_id"], row["total_cents"], row["created_at"]])
            last_id = row["id"]
            if last_id % 5000 == 0:
                save_checkpoint(checkpoint_path, last_id)
    save_checkpoint(checkpoint_path, last_id)

def load_checkpoint(path):
    try:
        with open(path) as f:
            return json.load(f)["last_id"]
    except FileNotFoundError:
        return 0

def save_checkpoint(path, last_id):
    with open(path, "w") as f:
        json.dump({"last_id": last_id}, f)

Opening the output file in append mode means a restart continues the same CSV file instead of overwriting it. The checkpoint is written every 5,000 rows rather than every row, trading a small amount of possible re-work on restart for far fewer disk writes during the normal run.

Glossary

18 terms

`__slots__`: A class attribute that replaces each instance's `__dict__` with a fixed set of named slots, cutting per-instance memory and blocking new attributes from being added later.
Closure: A function that remembers variables from the scope it was defined in, even after that scope has finished running. It is how a factory function can return a customized function.
Context manager: An object with `__enter__` and `__exit__` methods, used with `with` to guarantee cleanup — closing a file or releasing a lock — even if the block raises.
Coroutine: A function defined with `async def` that can pause at an `await` and let other code run, all on a single thread. It is the building block of `asyncio`.See also: Event loop
Decorator: A function that takes another function and returns a wrapped version of it, applied with `@name` above a definition. It adds behavior — logging, caching, retries — without changing the original function's code.
Descriptor: An object that customizes attribute access by defining `__get__`, `__set__`, or `__delete__`. Properties, methods, and `staticmethod` are all built from descriptors.
Duck typing: Python checks whether an object has the method or attribute you call, not what class it is. If it walks like a duck and quacks like a duck, your code treats it as one.
Dunder method: A method named with double underscores on both sides — `__init__`, `__len__`, `__eq__` — that Python calls automatically for a language feature like construction, `len()`, or `==`.
Event loop: asyncio's scheduler. It runs one coroutine until it hits an `await` on something not yet ready, then switches to another — this is what makes async concurrent without threads.See also: Coroutine
Generator: A function containing `yield` that produces values one at a time instead of building a full list in memory. Calling it returns an iterator that resumes from where it last paused.
GIL (Global Interpreter Lock): A lock in CPython that lets only one thread run Python bytecode at a time. It is why CPU-bound work does not speed up with threads, but I/O-bound work still can — the lock releases during a blocking call.
Idempotent: An operation that produces the same result no matter how many times you run it. Setting an order's status to "shipped" is idempotent; incrementing a counter is not.
Immutable: An object whose value cannot change after creation — strings, tuples, ints, and frozensets. Any "modification" creates a new object instead.See also: Mutable
LEGB: The order Python searches for a name: Local, Enclosing, Global, Built-in. It stops at the first scope where the name exists.
Metaclass: The class of a class — usually `type`. It controls how a class itself is created, which is how libraries like Django inject behavior into every subclass.
Monkey patching: Replacing or adding an attribute on a class or module at runtime, outside its own source file. Common in tests as mocking, and risky in production code, since it changes behavior invisibly to anyone reading the original definition.
MRO (Method Resolution Order): The order Python searches parent classes for a method or attribute in multiple inheritance, computed by the C3 linearization algorithm. `ClassName.__mro__` shows it directly.
Mutable: An object whose contents can change after creation — lists, dicts, and sets. The object keeps the same identity while its value changes.See also: Immutable

Frequently asked

10 questions

Why is Python slower than Java or Go?

CPython compiles to bytecode and runs it on a virtual machine, with dynamic typing checked on every operation — a name lookup, a type check, and a dispatch happen on every line that a compiled language resolves once. For CPU-bound loops this costs 10-50x. For I/O-bound work — web requests, database queries — the difference mostly disappears, because the program spends its time waiting, not computing.

Does the GIL mean Python cannot use multiple CPU cores?

For a single process, yes — only one thread runs Python bytecode at a time. Two ways around it: `multiprocessing` runs separate processes, each with its own GIL, for CPU-bound work; threads still help for I/O-bound work, because the GIL releases during a blocking call like a network read.

When do I use a list, tuple, set, or dict?

A list for an ordered, changeable sequence. A tuple for a fixed, ordered group — especially one you use as a dict key or unpack by position. A set when you need uniqueness or fast membership tests and order does not matter. A dict when you look values up by a key rather than a position.

When should I write a class instead of a function?

Reach for a class when you need to bundle state with the behavior that acts on it, or need more than one independent instance of that state. If a function only transforms its inputs into an output with no memory between calls, it does not need a class.

Which packaging tool should I use — pip, Poetry, or uv?

`pip` with a virtual environment (`venv`) is the standard library baseline and always works. Poetry and `uv` add a lock file, so every install reproduces the exact versions that were tested — worth it once a project has more than a few dependencies. `uv` does the same job as Poetry an order of magnitude faster.

Are type hints worth adding to a Python project?

Yes, once a project is bigger than a script. Python does not enforce them at runtime — a tool like `mypy` or `pyright` does, catching a class of bug (a wrong argument type, `None` where a value was assumed) before it reaches a test. Editors also use them for autocomplete and refactoring.

asyncio, threading, or multiprocessing — which one?

`asyncio` for many concurrent I/O-bound tasks — thousands of open connections — with cooperative code. `threading` for a handful of I/O-bound tasks using libraries not written for `async`. `multiprocessing` for CPU-bound work, since it is the only one of the three that uses more than one core.

What does a leading underscore on a name mean, like `_cache`?

It is a convention, not enforcement: it tells other developers this name is internal and may change without notice. Python does not block access to it. A double leading underscore (`__cache`) does trigger name mangling inside a class, which is a different, narrower mechanism aimed at avoiding subclass name collisions.

Why am I getting a circular import error?

Two modules are importing from each other, and Python has not finished executing the first one when the second tries to use something from it. Fix it by moving the shared code to a third module both can import, or by moving the import inside the function that needs it so it runs later, after both modules have finished loading.

Why did my function's default list argument keep old values from a previous call?

A default argument value is created once, when the function is defined, not on every call. If that default is a mutable object like a list, every call that omits its own argument shares and mutates the same list. Use `None` as the default and create the list inside the function body instead.