beginnerintermediateadvanced

Django · Performance Debugging

Django Performance Debugging

Split the duration, read the signature, then pick the tool that answers your question

Split one duration into four numbers firstdatabase, external, CPU, middleware — the layers look identical from a single total
db time and query COUNT point at opposite fixes431 fast queries is an N+1; 3 slow ones is an index
Healthy EXPLAIN + slow production = contentionthe lock wait is counted inside the statement duration
Memory kills the process, it does not slow itso the requests that die are rarely the guilty one
py-spy attaches to a live worker; the toolbar never shipsDebug Toolbar needs DEBUG, which changes far more than the toolbar

1.Locating the time in a slow request

Start by splitting one number into four. A request takes 2,400 ms; how much of that was the database, how much was waiting on other services, how much was your own Python, and how much happened before your view was even called? Middleware is the part people forget: every entry in `MIDDLEWARE` runs on every request, in order, on the way in and on the way out — so one middleware doing a database lookup per request adds its cost to every endpoint, including the health check. Until the 2,400 is broken up, any fix is a guess.

Key features

Every middleware runs on every request, both inbound and outbound — its cost lands on all endpoints, including health checks
Middleware ordering is a dependency graph, not a preference; reordering to "put the cheap one first" can break authentication or CSRF
Session and authentication middleware issue queries before your view runs, so "3 queries in the view" is rarely the request total
Split total into `db_ms`, `external_ms`, `cpu_ms` and middleware time before opening any file
A p95 does not tell you the shape — look at the distribution, or you cannot tell "uniformly slow" from "a bad tail"
Attach the numbers to individual slow requests; aggregates cannot be drilled into after the fact

Reading the four numbers

Reading the four numbers
Pattern	Means	Go to
`db_ms` high, `db_queries` high	N+1 or per-row work	the view's loops and the serializer
`db_ms` high, `db_queries` low	one genuinely slow query	`EXPLAIN`, indexes
`external_ms` high	a third party inside your request	timeouts, then move it to a task
`cpu_ms` high	your own computation	`cProfile` on the view, or `py-spy` in production
all four low, total high	time before or after the view	middleware, or the WSGI/ASGI server queue

Where 2,400 ms actually went — the same request, broken into spans

0 ms
Request accepted by the worker
anything before this is server queueing, not your code
0–40 ms
Middleware, inbound
session read + user lookup + feature flags = 3 queries before the view
40–2,250 ms
The view
db 2,210 ms across 431 queries — the count says N+1, not a slow query
2,250–2,380 ms
Serialization
invisible in the SQL log; shows up as cpu_ms
2,380–2,400 ms
Middleware, outbound
compression, headers — runs on every response too
after
What a single duration told you
nothing: the same 2,400 ms could have been any one of these spans

Command

python

MIDDLEWARE = [...]      # every entry runs on EVERY request, in and out
# cost here is multiplied by your total request rate, not by the endpoint's

Refuse to open a file until one duration is four numbers: database, external, CPU and — the one people forget — middleware. Every entry in `MIDDLEWARE` runs on every request in both directions, so a query in there is the most expensive query in the codebase; the tell is a floor under every endpoint, including ones that do nothing. And always read p50 next to p95 and p99, because uniform slowness and a bad tail look identical in a single percentile and need opposite investigations.

2.N+1 queries, connection exhaustion, and lock contention

Three database problems produce three different signatures. **N+1**: query count grows with row count, every query fast. **Connection exhaustion**: requests fail rather than slow down, with `too many clients already`, and it usually follows a scale-up. **Lock contention**: a small number of requests wait a long time on rows other requests hold, and the waiting is invisible in query duration until you look at what is blocking what. `EXPLAIN` answers a fourth question — why one query is slow — and it is the only one of the four that an index can fix.

Key features

N+1 signature: queries grow with rows, each one fast — `assertNumQueries` at two data sizes proves it
Exhaustion signature: errors rather than slowness (`FATAL: sorry, too many clients already`), usually right after adding workers
Contention signature: the same query is fast alone and slow under load, with a healthy `EXPLAIN` plan
`QuerySet.explain()` runs the database's own `EXPLAIN`; pass `analyze=True` on PostgreSQL to execute and get real row counts
In a plan, compare estimated rows against actual — a large gap means the planner is working from stale statistics
A transaction holds its locks until it commits, so anything slow *inside* `atomic()` is a contention source, especially an HTTP call

Four database symptoms, four different fixes

Four database symptoms, four different fixes
Signature	Diagnosis	Fix	What does NOT help
count grows with rows	N+1	`select_related` / `prefetch_related`	indexes, a bigger database
`too many clients already`	connection exhaustion	fewer workers, pooler, `CONN_MAX_AGE`	more workers
fast alone, slow under load	lock contention	shorten the holding transaction	`EXPLAIN`, indexes
one query always slow	plan or index problem	read `EXPLAIN`, add an index	caching around it

Narrowing a database symptom in four checks

1. Does the query count grow with the data?

Run the same request against 10 rows and 100 rows. If the count moves, stop here — it is an N+1, and nothing else on this list applies.

2. Are requests failing, or merely slow?

Failures naming clients or connections mean exhaustion, which is a worker-count problem upstream of the database, not a query problem.

3. Is the query fast when it runs alone?

Same statement, same parameters, on an idle system. Fast alone and slow under load is contention — the wait is counted inside the query duration and looks exactly like slowness.

4. Only now, read the plan

A sequential scan over a large table, or a large gap between estimated and actual rows, is what an index or fresh statistics can fix. Reaching this step first is what wastes the afternoon.

Command

python

qs.explain()                 # the plan
qs.explain(analyze=True)     # PostgreSQL: actually runs it, reports real rows

Sort by what grows. Query count growing with rows is an N+1 that no index fixes. Connections growing with workers is exhaustion, fixed upstream of the database. Waiting that grows with concurrency while the query stays constant is lock contention — and its tell is a perfect `EXPLAIN` plan beside a slow production duration, because the wait is counted inside the statement. Only a query that is consistently slow on its own deserves the plan-and-index treatment. Keep `atomic()` blocks short and free of network calls: a lock lives until commit, not until the statement ends.

3.High memory, huge payloads, and excessive serialization

These three are usually the same incident seen from three angles. An endpoint loads every row, builds a model instance for each, serializes them all into one response, and the process runs out of memory. The distinguishing evidence is that memory problems do not show up as slow requests — the process is killed, and the requests that die are whichever ones happened to be in flight. `tracemalloc` is the standard library tool for finding which lines allocated the memory: start it, take a snapshot, compare it to a later one, and read the top allocating lines.

Key features

Memory failures kill the process rather than slowing it, so the failing requests are usually not the guilty one
`tracemalloc.start()` then `Snapshot.compare_to(previous, "lineno")` names the lines that allocated between two points
`tracemalloc` costs real overhead — enable it deliberately on one process, not fleet-wide
Model instances are far heavier than the row: `.values()` skips instantiation, `.only()` shrinks it
`.iterator(chunk_size=…)` streams rather than materialising; `list(qs)` and `len(qs)` both hold everything
Response size is a separate budget from row count — cap it, and prefer a download job over a giant synchronous payload

Three quantities, three fixes

Three quantities, three fixes
Quantity	Grows with	Fix
rows fetched	the filter you did not apply	paginate; filter in the database
objects held	rows × fields per instance	`.values()`, `.only()`, `.iterator()`
bytes returned	rows × serialized fields	page the response, or export asynchronously
peak during serialization	building the whole payload before sending	`StreamingHttpResponse`

The same export, before and after — where each megabyte went

Materialised

Every row fetched, whatever the table size
One `Order` instance per row, all fields
The whole payload built in memory before the first byte is sent
Peak memory ≈ instances + serialized payload, at once
Failure mode: the container is killed, taking every other request

Streamed

Rows arrive in chunks; the database keeps the cursor
Dicts, not model instances
Bytes leave as they are produced
Peak memory ≈ one chunk, whatever the row count
Failure mode: none new — it is flat in the data size

Command

python

tracemalloc.start()
before = tracemalloc.take_snapshot()
...
for stat in tracemalloc.take_snapshot().compare_to(before, "lineno")[:10]:
    print(stat)

Memory failures kill the process rather than slowing it, so the requests that die are rarely the guilty one — diagnose from restarts, not from exceptions. Track rows fetched, objects held and bytes returned as three separate numbers: `.values()` removes the instance copy, `.iterator()` removes the all-at-once property, and streaming removes the payload copy. Use `tracemalloc` to name the allocating lines, deliberately and on one process, then turn it off. And never treat a slice as a bound on cost — cap the response in bytes, or hand genuinely large exports to a background job.

4.Slow dependencies, and the right tool for each symptom

Three dependencies commonly go slow and each has a distinct tell. **Redis latency**: everything that touches the cache gets slower at once, and cache reads start costing more than the database reads they were meant to replace. **Worker backlog**: nothing is slow at all — the queue depth grows, and users see stale results because their job has not run yet. **A slow external API**: time concentrates in one outbound call, and the fix is a timeout plus moving it out of the request. Then pick the tool by where you are: Debug Toolbar locally, `cProfile` for a reproducible path, py-spy for a live process you must not restart, and an APM for the fleet.

Key features

Redis latency shows as everything cache-touching slowing together — measure the cache call itself, not only the endpoint
A worker backlog produces no slow requests: the signal is queue depth and job age, which are invisible in request metrics
A slow external API concentrates time in one span; the fix is a timeout first, then moving the call out of the request
Django Debug Toolbar is development-only — it needs `DEBUG` and `INTERNAL_IPS`, and exposes settings and SQL
`cProfile` counts every call, so it inflates call-heavy code; use it on a reproducible path, not on live traffic
py-spy samples another process from outside — no restart, no code change — which makes it the tool for a wedged production worker

The tool for each question

The tool for each question
Question	Tool	Where it runs	Its distortion
which queries does this page run?	Django Debug Toolbar	local only	adds overhead; never enable in production
did the query count change?	`assertNumQueries`	CI	none — it is an assertion, not a measurement
which function is expensive here?	`cProfile`	local / a script	inflates call-heavy code
what is this process doing now?	py-spy	production, live	sampling: rare frames may be missed
where does time go across the fleet?	APM	production, always	sampled traces; per-request detail is thin
what happens under concurrency?	load testing	staging	reveals problems rather than explaining them

Choose by where you are and how wide the question is

the whole systemone line or function

local / reproduciblelive production traffic

cProfile: per-function costs on a path you can re-run; distorts call-heavy code
Django Debug Toolbar: queries, duplicates and templates for one page — development only
assertNumQueries: not a measurement: a regression gate that runs in CI forever
py-spy: attach to a running worker, no restart — the tool for a wedged process
APM traces: the only view that shows patterns across requests and services
load testing: creates the conditions — contention and backlog do not exist in one request

Command

bash

python -m cProfile -s cumtime manage.py rebuild_report 2026-09 | head -30
py-spy dump --pid $(pgrep -f "gunicorn: worker" | head -1)

Give every dependency its own span: Redis latency hides inside whatever endpoint called the cache, and a worker backlog produces no slow request at all — its signals are queue depth and job age. Pick the tool by the question and where you are: Debug Toolbar locally (never in production, since it needs `DEBUG`), `cProfile` on a reproducible path, py-spy on a live worker you must not restart, an APM for patterns across the fleet, and load testing to create the conditions the others need. Profile before benchmarking — a component's share of the total is the ceiling on any improvement to it.

Spotted a mistake on this page? Tell us.

Read Sheet