beginnerintermediateadvanced

Django · Observability

Observability

Alert on metrics, debug in traces, confirm in logs — joined by one request id

Metric cost grows with label cardinalitylabel with the URL route, never the path — the path contains ids
One request id, injected by a logging filternot passed by hand, or the lines you forget are the interesting ones
A queue backlog produces no slow requestdepth and oldest-job age must be emitted on a schedule
Saturation leads, latency lagspools do not degrade gently — they are fine, then they queue
Alert on symptoms, never on causesif the right response to a page is to close it, it was a dashboard panel

1.Logs, metrics, and traces — three questions, not three formats

The three signals are not three ways of writing down the same thing. A **log** is a record of one event, with as much detail as you like — it answers "what happened in this request". A **metric** is a number aggregated over time, cheap to keep and cheap to query — it answers "is this happening more than usual". A **trace** follows one request across services and shows where its time went. You need all three because each is bad at the others' job: you cannot alert on logs affordably, you cannot debug a specific failure from a metric, and neither shows you which downstream service was slow.

Key features

Metric cost grows with **label cardinality**, not traffic — never label with a user id, an order id, or a raw URL path
Log cost grows with volume, and their value is per-event detail — the opposite trade-off
Traces are the only signal that crosses a service boundary, and they are normally sampled
Alert on metrics, debug in traces, confirm in logs — that ordering is what keeps each cost sane
A shared `request_id` on all three is what makes them one system instead of three tools
Django logs `django.request` at ERROR for 5XX and WARNING for 4XX, with `status_code` and `request` attached to the record

Which signal answers which question

Which signal answers which question
Question	Signal	Cost grows with
Is something wrong right now?	metrics	label combinations
Is it worse than yesterday?	metrics	label combinations
Where did this request spend its time?	traces	sample rate
Which service in the chain was slow?	traces	sample rate
What exactly happened to this user?	logs	event volume
What was the stack trace?	error tracking	unique issues, not occurrences

Three signals, three questions — and the field that joins them

Alerts fire on metrics because metrics are cheap to query; you debug in traces because only traces cross service boundaries; you confirm in logs because only logs hold the detail.

Command

python

metrics.timing("db.query_ms", ms, tags={"view": view_name})   # bounded labels
log.warning("payment declined", extra={"order_id": order.id, "request_id": rid})

Alert on metrics, debug in traces, confirm in logs — the ordering follows from what each one costs. Metric cost grows with label cardinality, so labels must come from a small fixed set: use the URL route, never the path; put ids on trace attributes and in structured log fields instead, where they are free. Traces are the only signal that crosses a service boundary. And give all three the same `request_id`, because that single shared field is the difference between one system and three search boxes.

2.Request ids, correlation ids, and error tracking

A **request id** identifies one HTTP request. A **correlation id** identifies one piece of work across everything it touches — the request, the Celery task it enqueued, the call it made to another service, and the webhook that came back later. In practice you generate a request id if the caller did not send one, put it on every log line, and pass it along to every task and outbound call. **Error tracking** is the third piece: a service that groups identical exceptions into one issue with a count, rather than making you find them among the logs.

Key features

Generate a request id at the edge if the caller did not supply one; propagate an existing one if they did
Attach it to every log record automatically, with a logging filter — not by remembering to pass it
Pass it explicitly to Celery tasks and outbound HTTP calls; nothing propagates it for you across a process boundary
Return it in a response header so a user can quote it in a support ticket
Error tracking groups by exception type and shape — a unique message per occurrence destroys the grouping
Store the id on long-lived records (an order, a webhook event) so a trace is possible days later

How the id crosses each boundary

How the id crosses each boundary
Boundary	Carried as	If you forget
inbound HTTP	`X-Request-ID` header, or generated	every chain starts anonymous
log records	a filter injecting it onto every record	only the lines you remembered have it
Celery task	an explicit argument or a task header	the task's logs are unlinkable to the request
outbound HTTP	`X-Request-ID` on the request	the other service's logs cannot be joined to yours
inbound webhook	the provider's event id, stored	a late callback cannot be tied to its origin
the response	`X-Request-ID` header	a user's bug report has nothing to search on

One checkout, five processes, one id

browser

django

celery

payments API

error tracker

1POST /checkout/ (no X-Request-ID)
2generate rid = 7f3a…at the edge, before anything else runs
3POST /charges X-Request-ID: 7f3a…their logs can now be joined to yours
4402 declined
5PaymentDeclined { request_id: 7f3a… }grouped by type, not by message
6notify_customer.delay(order_id, request_id="7f3a…")nothing propagates it automatically
7logs carry rid = 7f3a…
8402 X-Request-ID: 7f3a…the id the support ticket will quote

Command

python

rid = request.headers.get("X-Request-ID") or uuid.uuid4().hex   # propagate or generate

Create the id at the edge — propagate the caller's if there is one, generate one otherwise — and inject it onto log records with a *filter*, so coverage does not depend on anyone remembering, and Django's own `django.request` records get it too. Carry it explicitly across boundaries nothing crosses for you: Celery arguments, outbound headers, and columns on records you may need to trace days later. Return it in a response header so "it broke" becomes a searchable id. And keep exception types and messages stable, putting variable data in context — a unique message per occurrence destroys the grouping that makes error tracking useful.

3.The signals worth measuring

Six signals cover most of what goes wrong in a Django service: **API latency**, **error rate**, **saturation** (how full the thing is), **database latency**, **cache hit rate**, and **queue metrics** — depth and the age of the oldest job. The last one is the one people miss, because a queue backlog produces no slow request and no error: everything looks healthy while work silently falls further behind. Measure latency as percentiles rather than averages, and measure saturation for every fixed-size resource you have: workers, connections, and queue capacity.

Key features

Latency as p50/p95/p99 — an average hides the tail, and the tail is what users report
Error rate as a *ratio* of requests, so it stays meaningful when traffic changes
Saturation for every fixed-size pool: workers busy, connections used, queue slots — these move before latency does
Queue depth **and** oldest-job age: depth alone cannot tell a draining backlog from a stuck one
A backlog produces no slow request and no error — request metrics are blind to it entirely
Cache hit rate is what makes "we added caching" a checkable claim rather than an assertion

The six signals, and what each one catches first

The six signals, and what each one catches first
Signal	Measure as	Catches
API latency	p50 / p95 / p99 per route	a slow endpoint, and whether it is uniform or a tail
Error rate	errors ÷ requests, by status class	a bad deploy, a failing dependency
Saturation	busy ÷ capacity, per pool	the resource about to run out — before latency moves
DB latency	query time and query count	an N+1 (count) vs a plan problem (time)
Cache hit rate	hits ÷ (hits + misses)	a cache that is costing more than it saves
Queue depth + age	both, per queue	work falling behind while every request looks fine

Group the signals by the question they answer first

What users feel

API latency p50/p95/p99

per route; an average hides the tail

Error rate as a ratio

by status class, so traffic changes do not distort it

What is filling up

Workers busy ÷ capacity

a pool is fine, then it queues — there is no gentle slope

Connections used ÷ max

the ceiling refuses rather than degrades

Memory per process

the one that terminates instead of slowing

What is falling behind

Queue depth

how much work is waiting

Oldest-job age

the number that separates draining from stuck

Task failure and retry rate

a task retrying forever raises neither of the above

What your fixes claim

DB time AND query count

the two point at opposite fixes

Cache hit rate

below a decent rate a cache is a net cost

Command

python

metrics.timing("http.latency_ms", ms, tags={"route": route, "status_class": "2xx"})
metrics.gauge("db.connections_used", used)     # saturation, not latency

Six signals: API latency as percentiles, error rate as a ratio, saturation for every fixed-size pool, database time *and* query count, cache hit rate, and queue depth *plus* oldest-job age. Saturation leads and latency lags, because pools do not degrade gently — they are fine, then they queue. And the queue signals are the ones that must be emitted on a schedule rather than derived from requests, since a backlog produces no slow request and no error: it is the one incident your request dashboards cannot see even in principle.

4.Alerts, dashboards, and the ecosystem

An alert should wake someone only when a human needs to act now. Anything else belongs on a dashboard, which is for looking at deliberately. The test that keeps this honest is simple: if an alert fires and the correct response is to close it, it should not have been an alert. Alert on **symptoms** users feel — error rate, latency, queue age — rather than on causes like CPU, because there are many causes and only a few symptoms. The common tools divide by job: Sentry for errors, Prometheus for metrics, Grafana for dashboards, OpenTelemetry as the vendor-neutral way to emit traces and metrics, and a cloud APM bundling several of these.

Key features

Alert on symptoms (error rate, latency, queue age), not on causes (CPU, memory) — fewer rules, more real coverage
Every threshold needs a duration: "above X **for** N minutes" filters out normal spikes
If the right response to an alert is to close it, it should have been a dashboard panel
Each alert should name what to do, or link to something that does
Sentry groups exceptions into issues · Prometheus stores queryable time series · Grafana visualises them · OpenTelemetry is the vendor-neutral way to emit traces and metrics · a cloud APM bundles several
The value is in the links between tools, and the request id is what makes those links possible

Alert or dashboard?

Alert or dashboard?
Signal	Alert?	Why
error rate > 2% for 5 min	yes	users are affected now, and someone must look
p95 latency > 2s for 10 min	yes	a symptom, and sustained rather than a spike
oldest queued job > 5 min	yes	invisible everywhere else, and worsens silently
DB connections > 80% of max	yes	saturation leads — there is time to act before it refuses
CPU > 80%	no	a cause, not a symptom; often entirely fine
one task failed and retried	no	retries are the design working; alert on the retry rate
cache hit rate	no	a dashboard number for judging a change

From a page to a root cause, without changing tools blindly

1. The alert fires on a symptom

A metric crossed a threshold and stayed there. It says what is wrong for users, not what caused it, and it links to the dashboard for this service.

2. The dashboard says which layer

Latency, error rate, saturation and queue panels side by side. The point is not detail — it is deciding whether this is database, dependency, queue or capacity.

3. A trace shows where the time went

One sampled request, broken into spans across services. This is the only view that crosses a process boundary and names the slow hop.

4. Logs and the error tracker confirm it

Search by the request id from the trace. The error tracker groups the exception into one issue with a count and a first-seen time, which answers "is this new?".

Command

text

alert: <symptom metric> <comparison> <threshold> for <duration>
       + a link to the dashboard, and a runbook line saying what to do

Alert on symptoms users feel — error rate, latency, queue age, saturation — never on causes like CPU, because there are many causes and few symptoms. Put a duration on every threshold, and make each alert name what to do. Everything else is a dashboard panel: if the right response to a page is to close it, it should not have been a page. Keep the tool set small and *linked* — alert to dashboard to trace to logs, joined by the request id — and remember Django already emits much of what you need, with `django.request` logging 5XX at ERROR and 4XX at WARNING.

Spotted a mistake on this page? Tell us.

Read Sheet