beginnerintermediateadvanced

Django · Section 77

Observability

Level: intermediate
Read: 40 min
Concepts: 4

Logs, metrics and traces are not three formats for the same information — they are three questions with three different cost models, and choosing between them by cost is what keeps a monitoring system affordable. A metric is pre-aggregated, so its cost grows with the number of distinct label combinations rather than with traffic; that is what makes it cheap to alert on, and also why a user id or a raw URL path in a label is the classic way to bring a metrics backend down. A log costs in proportion to volume and pays you back in per-event detail, so it is where you go once you already know something is wrong. A trace is the only signal that crosses a process boundary, and it is normally sampled. The working order follows: alert on metrics, debug in traces, confirm in logs. What turns those three tools into one system is a single shared identifier. A request id is created at the edge — propagated if the caller sent one, generated otherwise — and injected onto every log record by a logging *filter* rather than passed by hand, because anything relying on call sites will be missing exactly the framework-emitted lines you need most. It then has to be carried explicitly across the boundaries nothing crosses for you: Celery arguments, outbound headers, and columns on records you may want to trace days later. Error tracking joins the same chain, and its whole value is grouping — which is destroyed by interpolating ids into exception messages. Then there are the numbers themselves. Six cover most Django failures: API latency as percentiles, error rate as a ratio, saturation for every fixed-size pool, database time *and* query count, cache hit rate, and queue depth plus oldest-job age. Saturation leads and latency lags, because a pool does not degrade gently — it is fine, and then it queues. And the queue signals are the ones that must be emitted on a schedule, because a backlog produces no slow request and no error, which makes it the one incident request dashboards cannot see even in principle. Finally, alerts and dashboards have opposite goals: alert on symptoms users feel, with a duration on every threshold and a runbook line on every rule, and let everything merely interesting live on a dashboard.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

Three signals, three cost models — alert on metrics, debug in traces, confirm in logs.
Metric labels must come from a small fixed set; ids belong on spans and log fields.
One request id, injected by a filter, carried explicitly across queue and HTTP boundaries.
Six signals worth emitting — and the queue ones need a scheduled emitter.
Alert on symptoms with a duration; everything else is a dashboard panel.

What you will be able to do

Choose the right signal for a question instead of logging everything
Trace one user action across a request, a task and a webhook
Emit the signals that reveal a backlog nothing else can see
Build an alert set small enough that people still trust it

What each layer answers during an incident, top to bottom

Alerts — "someone must act now"

symptoms only, with a duration and a runbook: error rate, latency, queue age, saturation

Dashboards — "which layer is it?"

latency, errors, saturation and queue panels side by side; read on purpose, never paged on

Traces — "where did the time go?"

the only signal that crosses a service boundary; sampled, and carries high-cardinality ids for free

Logs — "what exactly happened?"

structured, one record per event, every one carrying the request id via a logging filter

Error tracking — "is this new, and how often?"

groups occurrences into issues; stable exception messages are what make the grouping work

request_id — the field every layer shares

without it these are five tools; with it they are one path from alert to root cause

Three signals, three questions

What each one is good at, what each one costs, and why the order alert → trace → log is not arbitrary.

Logs, metrics, and traces — three questions, not three formats

coreintermediate

The three signals are not three ways of writing down the same thing. A **log** is a record of one event, with as much detail as you like — it answers "what happened in this request". A **metric** is a number aggregated over time, cheap to keep and cheap to query — it answers "is this happening more than usual". A **trace** follows one request across services and shows where its time went. You need all three because each is bad at the others' job: you cannot alert on logs affordably, you cannot debug a specific failure from a metric, and neither shows you which downstream service was slow.

Think of it as

Choose the signal by the question, and let cost decide the shape. Metrics are pre-aggregated, so the cost does not grow with traffic — it grows with the number of distinct label combinations, which is why putting a user id or a URL with an id in it into a metric label is the classic way to make a monitoring system fall over. That is the same property that makes them the right thing to alert on: querying "error rate over the last five minutes" is a cheap lookup, not a search. Logs are the opposite: cost grows directly with volume, and their value is detail — the parameters, the user, the exact exception. So logs are where you go once you already know something is wrong and want to know what, and that is also why they should be structured, since searching by field beats grepping for a substring. Traces sit between the two: they carry the shape of a request across process boundaries, which neither of the others can, and they are usually sampled because keeping every one is expensive. The thing that turns three separate tools into one system is a shared identifier. If the log line, the trace and the error report all carry the same `request_id`, an alert on a metric leads to a trace, which leads to the exact log lines, which lead to the stack trace. Without it you have three search boxes and no way to line up their answers, which in practice means people use whichever one they know and guess about the rest.

python

metrics.timing("db.query_ms", ms, tags={"view": view_name})   # bounded labels
log.warning("payment declined", extra={"order_id": order.id, "request_id": rid})

What we're doing: Emit all three from one request, with labels that stay bounded and a shared id that ties them together.

billing/services.pypython

def settle(order, request_id):
    with tracer.start_as_current_span("settle_order") as span:
        span.set_attribute("order.id", order.id)          # traces CAN carry high cardinality
        span.set_attribute("request.id", request_id)

        try:
            with tracer.start_as_current_span("payment.charge"):
                charge = payment_client.charge(order.total, timeout=(3.05, 10))
        except PaymentDeclined:
            metrics.increment("payments.result", tags={"result": "declined"})
            log.warning(
                "payment declined",
                extra={"order_id": order.id, "request_id": request_id, "amount": order.total},
            )
            raise

        metrics.increment("payments.result", tags={"result": "succeeded"})
        metrics.timing("payments.latency_ms", charge["elapsed_ms"])
        log.info(
            "payment settled",
            extra={"order_id": order.id, "request_id": request_id, "charge_id": charge["id"]},
        )
        return charge


# WRONG — every order id becomes its own metric series
metrics.increment(f"payments.order.{order.id}")
metrics.increment("payments.result", tags={"order_id": order.id, "user": user.email})

2–4: The span is the only place a high-cardinality id belongs as an attribute. Traces are stored per request, so an order id there costs nothing extra — unlike the same id on a metric.
10: One metric name, one bounded label: `result` takes a handful of values whatever the traffic. This is the shape that can be alerted on.
11–14: The log carries the detail the metric deliberately does not, plus the `request_id` that lets someone jump from the metric to this exact line.
18–19: A counter and a timing from the same event. The counter answers "how often", the timing answers "how slow" — and neither can substitute for the other.
26–28: The two forms that break a metrics backend: an id in the metric *name*, and an id or an email in a *label*. Both create an unbounded number of series, and the second also puts personal data into a system that is rarely access-controlled like your database.

Why this works: The same event produces a cheap aggregate, a detailed record and a span — each carrying what it is good at, and all three carrying the id that lets you walk between them.

Putting an unbounded value in a metric label

Wrong

python

metrics.timing("http.request_ms", ms, tags={"path": request.path})
# /orders/1/  /orders/2/  /orders/3/ ... one series per order, forever

Better

python

metrics.timing("http.request_ms", ms, tags={"route": request.resolver_match.route})
# "orders/<int:pk>/" — one series per ROUTE, a bounded set

What you see: The metrics backend slows down, then starts dropping data or billing sharply more, and dashboards that used to load instantly time out. The cause is usually weeks old by the time it is noticed.

Why: A metrics system stores one time series per unique combination of name and labels, so a label whose values are unbounded creates unbounded series — this is called a cardinality explosion, and it degrades the whole backend rather than one dashboard. `request.path` contains ids; `resolver_match.route` is the URL *pattern*, which takes as many values as you have routes. The general rule is that a label value must come from a small, fixed set you can name. High-cardinality context still has two good homes: trace attributes and structured log fields.

Three signals, three questions — and the field that joins them

Alerts fire on metrics because metrics are cheap to query; you debug in traces because only traces cross service boundaries; you confirm in logs because only logs hold the detail.

Which signal answers which question

Which signal answers which question
Question	Signal	Cost grows with
Is something wrong right now?	metrics	label combinations
Is it worse than yesterday?	metrics	label combinations
Where did this request spend its time?	traces	sample rate
Which service in the chain was slow?	traces	sample rate
What exactly happened to this user?	logs	event volume
What was the stack trace?	error tracking	unique issues, not occurrences

Together

python

metrics.increment("orders.settled", tags={"result": "declined"})   # low cardinality
log.info("order settled", extra={"order_id": order.id, "request_id": rid})  # detail

Remember: Alert on metrics, debug in traces, confirm in logs — the ordering follows from what each one costs. Metric cost grows with label cardinality, so labels must come from a small fixed set: use the URL route, never the path; put ids on trace attributes and in structured log fields instead, where they are free. Traces are the only signal that crosses a service boundary. And give all three the same `request_id`, because that single shared field is the difference between one system and three search boxes.

Tying it together

The id that crosses every boundary, and the error tracking it feeds.

Request ids, correlation ids, and error tracking

coreintermediate

A **request id** identifies one HTTP request. A **correlation id** identifies one piece of work across everything it touches — the request, the Celery task it enqueued, the call it made to another service, and the webhook that came back later. In practice you generate a request id if the caller did not send one, put it on every log line, and pass it along to every task and outbound call. **Error tracking** is the third piece: a service that groups identical exceptions into one issue with a count, rather than making you find them among the logs.

Think of it as

The problem these solve is that a single user action is not a single thing in your systems. A checkout is a request, three tasks, two outbound calls and a webhook that arrives four seconds later, and each of those writes its own logs from its own process. Without a shared id you cannot reconstruct the sequence at all — you can only search by timestamp and hope. The id has to be created at the very edge, before anything else happens, and then travel by whatever mechanism each boundary provides: a header for HTTP, a task argument or header for the queue, a column for anything you persist and might need to trace later. The rule that makes this work in practice is "generate if absent, propagate if present": accept an incoming id when a trusted caller sends one so the whole chain shares theirs, and mint one otherwise so there is never a gap. Error tracking is a different shape of tool and worth understanding as such. Its value is grouping: a hundred thousand occurrences of one exception become one issue with a count and a first-seen date, which turns "the logs are full of errors" into a list of distinct problems ordered by how much they matter. That grouping is also its weakness — if you attach the message to the exception in a way that makes every occurrence unique, the grouping breaks and you are back to a stream. So keep the exception type and message stable and put the varying parts in structured context.

python

rid = request.headers.get("X-Request-ID") or uuid.uuid4().hex   # propagate or generate

What we're doing: Set the id once at the edge, get it onto every log record without touching call sites, and carry it into a task.

observability/request_id.pypython

import contextvars, logging, uuid

request_id_var = contextvars.ContextVar("request_id", default="-")


class RequestIDFilter(logging.Filter):
    """Puts the current id on EVERY record — including Django's own."""

    def filter(self, record):
        record.request_id = request_id_var.get()
        return True


class RequestIDMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        rid = request.headers.get("X-Request-ID") or uuid.uuid4().hex
        token = request_id_var.set(rid)
        request.request_id = rid
        try:
            response = self.get_response(request)
            response["X-Request-ID"] = rid          # so a user can quote it
            return response
        finally:
            request_id_var.reset(token)


LOGGING = {
    "version": 1,
    "filters": {"request_id": {"()": "observability.request_id.RequestIDFilter"}},
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "filters": ["request_id"],
            "formatter": "json",
        },
    },
    "root": {"handlers": ["console"], "level": "INFO"},
}


@shared_task(bind=True)
def notify_customer(self, order_id, request_id="-"):
    request_id_var.set(request_id)                  # re-establish it in the worker
    log.info("notifying customer", extra={"order_id": order_id})

6–11: A logging filter is what makes this reliable. Every record passing through the handler gets the field, including records from `django.request` and from libraries that know nothing about your middleware.
20: "Propagate if present, generate if absent" in one line. Accepting an inbound id lets a whole call chain share one; generating otherwise means no request is ever anonymous.
26–27: Resetting the context variable in a `finally` matters under reuse: a worker thread that keeps a stale id will label the *next* request with the previous one's.
30: The filter is attached to the handler, so the field is injected once for the whole application rather than at each call site.
41–44: The queue boundary is manual. The id is an ordinary task argument and is re-established in the worker, because a Celery process shares no context with the request that enqueued it.

Why this works: The id is created once, injected automatically into every log record, returned to the caller and carried explicitly across the one boundary that cannot carry it implicitly — so a single search reconstructs the whole chain.

Passing the id by hand to each log call

Wrong

python

log.info("charging card", extra={"request_id": request.request_id})
log.info("charge complete")            # forgot it here
# and Django's own django.request logs never have it at all

Better

python

# one filter on the handler; every record gets it, including Django's
"filters": {"request_id": {"()": "observability.request_id.RequestIDFilter"}}

What you see: Half the log lines carry the id and half do not, and the missing half is reliably the interesting one — the unhandled exception logged by the framework rather than by your code.

Why: Anything relying on every call site remembering will be incomplete, and the gaps fall where you have the least control: Django's own loggers, third-party libraries, and the paths nobody thought about while writing the happy case. A filter runs on every record that reaches the handler, so coverage is a property of the configuration rather than of anyone's discipline. It also means the id is present on `django.request`'s ERROR records for 5XX responses, which are exactly the lines you want to find later.

One checkout, five processes, one id

browser

django

celery

payments API

error tracker

1. POST /checkout/ (no X-Request-ID)
2. generate rid = 7f3a…at the edge, before anything else runs
3. POST /charges X-Request-ID: 7f3a…their logs can now be joined to yours
4. 402 declined
5. PaymentDeclined { request_id: 7f3a… }grouped by type, not by message
6. notify_customer.delay(order_id, request_id="7f3a…")nothing propagates it automatically
7. logs carry rid = 7f3a…
8. 402 X-Request-ID: 7f3a…the id the support ticket will quote

How the id crosses each boundary

How the id crosses each boundary
Boundary	Carried as	If you forget
inbound HTTP	`X-Request-ID` header, or generated	every chain starts anonymous
log records	a filter injecting it onto every record	only the lines you remembered have it
Celery task	an explicit argument or a task header	the task's logs are unlinkable to the request
outbound HTTP	`X-Request-ID` on the request	the other service's logs cannot be joined to yours
inbound webhook	the provider's event id, stored	a late callback cannot be tied to its origin
the response	`X-Request-ID` header	a user's bug report has nothing to search on

Together

python

response["X-Request-ID"] = request_id     # so "it broke" becomes a searchable id

Remember: Create the id at the edge — propagate the caller's if there is one, generate one otherwise — and inject it onto log records with a *filter*, so coverage does not depend on anyone remembering, and Django's own `django.request` records get it too. Carry it explicitly across boundaries nothing crosses for you: Celery arguments, outbound headers, and columns on records you may need to trace days later. Return it in a response header so "it broke" becomes a searchable id. And keep exception types and messages stable, putting variable data in context — a unique message per occurrence destroys the grouping that makes error tracking useful.

What to measure

Six signals — including the two that no request will ever produce for you.

The signals worth measuring

coreintermediate

Six signals cover most of what goes wrong in a Django service: **API latency**, **error rate**, **saturation** (how full the thing is), **database latency**, **cache hit rate**, and **queue metrics** — depth and the age of the oldest job. The last one is the one people miss, because a queue backlog produces no slow request and no error: everything looks healthy while work silently falls further behind. Measure latency as percentiles rather than averages, and measure saturation for every fixed-size resource you have: workers, connections, and queue capacity.

Think of it as

Think in terms of "what is full" and "what is falling behind", not only "what is slow". Latency and error rate describe requests, and they are what users feel directly — but they are lagging indicators of the resources underneath. Saturation is the leading one: worker pool utilisation, connection pool usage and queue depth all move before latency does, because a resource does not degrade smoothly as it fills; it is fine, and then it queues. That is why saturation deserves its own metrics rather than being inferred from latency, and why every fixed-size pool in the system should have a number showing how much of it is in use. The queue is the clearest case of a signal that no request-side metric can reveal. If jobs arrive faster than they are consumed, every request stays fast, no error is raised anywhere, and the only visible effect is that things happen later and later — an email that arrives an hour after signup, a report that is always yesterday's. Depth alone is not enough either, because a big queue that is draining quickly is fine and a small one that is stuck is not: the age of the oldest unprocessed job is the number that distinguishes them. Cache hit rate belongs in the same list for a related reason — it is the number that tells you whether a cache is helping, and without it "we added caching" is a claim nobody can evaluate. Finally, always percentiles: an average latency is dominated by the many fast requests and can look unchanged while the slowest tenth of users have an unusable experience.

python

metrics.timing("http.latency_ms", ms, tags={"route": route, "status_class": "2xx"})
metrics.gauge("db.connections_used", used)     # saturation, not latency

What we're doing: Emit the six signals, including the two that no request ever produces.

observability/signals.pypython

class RequestMetricsMiddleware:
    def __call__(self, request):
        start = time.monotonic()
        response = self.get_response(request)

        route = getattr(request.resolver_match, "route", "unmatched")   # bounded label
        metrics.timing(
            "http.latency_ms", (time.monotonic() - start) * 1000,
            tags={"route": route, "status_class": f"{response.status_code // 100}xx"},
        )
        metrics.increment("http.requests", tags={"route": route,
                                                 "status_class": f"{response.status_code // 100}xx"})
        return response


@shared_task
def emit_queue_metrics():
    """Runs on a schedule. NOTHING in the request path produces these."""
    with celery_app.connection_or_acquire() as conn:
        for queue in ("default", "emails", "exports"):
            depth = conn.default_channel.queue_declare(queue=queue, passive=True).message_count
            metrics.gauge("celery.queue.depth", depth, tags={"queue": queue})

    oldest = (
        TaskRecord.objects
        .filter(state="pending")
        .aggregate(oldest=Min("enqueued_at"))["oldest"]
    )
    if oldest:
        metrics.gauge("celery.queue.oldest_age_s", (timezone.now() - oldest).total_seconds())


def emit_saturation():
    """Every fixed-size pool gets a used-versus-capacity number."""
    metrics.gauge("db.connections_used", db_connections_in_use())
    metrics.gauge("db.connections_max", settings.DB_MAX_CONNECTIONS)
    metrics.gauge("web.workers_busy", workers_busy())
    metrics.gauge("web.workers_total", settings.GUNICORN_WORKERS)


# depth 40219 · oldest_age_s 1841 · http p95 60ms · error rate 0.1%
#   -> every request-side signal is healthy. The incident is only in the queue.

6–10: `resolver_match.route` is the URL pattern, so the label set is the size of your URLconf. Using `request.path` here would create one series per object id.
16–21: A scheduled task, because there is no request during which queue depth could be recorded. This is the structural reason backlogs go unnoticed.
24–29: Oldest-job age is the signal that makes depth interpretable: 40,000 messages draining in a minute is fine, and 200 messages that have not moved in an hour is not.
33–36: Used and capacity as two gauges rather than a precomputed ratio. Keeping both lets a dashboard show the headroom and lets an alert fire on either the fraction or the absolute number.
39–40: The line that justifies the whole concept: four healthy request-side numbers beside one signal that is on fire. Only the queue metrics see it.

Why this works: Request-derived signals cover what users feel, saturation gauges cover what is about to run out, and the scheduled queue emitter covers the class of incident that produces neither a slow request nor an error.

Monitoring average latency

Wrong

text

avg_response_time = 120ms      -> "the site is fast"

Better

text

p50 = 45ms · p95 = 2.0s · p99 = 31s
  -> most requests are fine and roughly 1 in 100 is unusable.
     The average is 120ms because the fast requests outnumber the slow ones.

What you see: Dashboards stay green while support tickets pile up from a consistent minority of users, and nobody can connect the two because the metric says everything is fine.

Why: An average is dominated by the bulk of the distribution, so a small population of very slow requests barely moves it — and that population is exactly who complains. Percentiles describe the shape instead: p50 tells you the common experience, p95 and p99 tell you the worst one people actually have. The gap between them is also diagnostic, since a large gap means something specific distinguishes the slow requests — a big tenant, a deep page, a cold cache — which is a different investigation from everything being uniformly slow.

Group the signals by the question they answer first

What users feel

API latency p50/p95/p99

per route; an average hides the tail

Error rate as a ratio

by status class, so traffic changes do not distort it

What is filling up

Workers busy ÷ capacity

a pool is fine, then it queues — there is no gentle slope

Connections used ÷ max

the ceiling refuses rather than degrades

Memory per process

the one that terminates instead of slowing

What is falling behind

Queue depth

how much work is waiting

Oldest-job age

the number that separates draining from stuck

Task failure and retry rate

a task retrying forever raises neither of the above

What your fixes claim

DB time AND query count

the two point at opposite fixes

Cache hit rate

below a decent rate a cache is a net cost

The six signals, and what each one catches first

The six signals, and what each one catches first
Signal	Measure as	Catches
API latency	p50 / p95 / p99 per route	a slow endpoint, and whether it is uniform or a tail
Error rate	errors ÷ requests, by status class	a bad deploy, a failing dependency
Saturation	busy ÷ capacity, per pool	the resource about to run out — before latency moves
DB latency	query time and query count	an N+1 (count) vs a plan problem (time)
Cache hit rate	hits ÷ (hits + misses)	a cache that is costing more than it saves
Queue depth + age	both, per queue	work falling behind while every request looks fine

Together

python

metrics.gauge("celery.queue.depth", depth, tags={"queue": "default"})
metrics.gauge("celery.queue.oldest_age_s", age, tags={"queue": "default"})

Remember: Six signals: API latency as percentiles, error rate as a ratio, saturation for every fixed-size pool, database time *and* query count, cache hit rate, and queue depth *plus* oldest-job age. Saturation leads and latency lags, because pools do not degrade gently — they are fine, then they queue. And the queue signals are the ones that must be emitted on a schedule rather than derived from requests, since a backlog produces no slow request and no error: it is the one incident your request dashboards cannot see even in principle.

Alerts, dashboards, and tools

Waking someone versus informing them, and what each tool in the ecosystem is actually for.

Alerts, dashboards, and the ecosystem

coreintermediate

An alert should wake someone only when a human needs to act now. Anything else belongs on a dashboard, which is for looking at deliberately. The test that keeps this honest is simple: if an alert fires and the correct response is to close it, it should not have been an alert. Alert on **symptoms** users feel — error rate, latency, queue age — rather than on causes like CPU, because there are many causes and only a few symptoms. The common tools divide by job: Sentry for errors, Prometheus for metrics, Grafana for dashboards, OpenTelemetry as the vendor-neutral way to emit traces and metrics, and a cloud APM bundling several of these.

Think of it as

Alerts and dashboards have opposite design goals, and treating them as the same thing at different urgencies is what produces alert fatigue. An alert interrupts a person, so its bar is "someone must do something within minutes" — which means it needs to be rare, actionable, and to point at what to do. A dashboard is read on purpose, so it can be dense, exploratory and full of things that are merely interesting. The most common failure is promoting dashboard-shaped information into pages: CPU above 80%, a single failed task, a spike that resolved itself. Each is defensible alone, and together they train people to dismiss notifications, which costs you the one alert that mattered. Alerting on symptoms rather than causes falls out of the same reasoning: high CPU may be perfectly fine, and there are dozens of ways to be broken without it, so "error rate above 2% for five minutes" covers far more real failures with far fewer rules. Add duration to every threshold, because instantaneous spikes are normal and the sustained version is the incident. On tooling, resist collecting one of everything. Each tool answers one question well — errors grouped into issues, time series you can query, dashboards over those series, a vendor-neutral emission standard — and the value comes from linking them so an alert leads to a dashboard, which leads to a trace, which leads to the log lines, all joined by the request id. A tool nobody opens during an incident is not observability; it is a bill.

text

alert: <symptom metric> <comparison> <threshold> for <duration>
       + a link to the dashboard, and a runbook line saying what to do

What we're doing: A small alert set that covers real failures, plus the Django-side configuration that feeds it.

observability/alerts.md + settings.pypython

# ---- The whole alert set for one service -------------------------------
#
# 1. error_rate{service="shop"} > 0.02              for 5m    -> page
#      "users are seeing failures"  · runbook: recent deploys, then dependencies
# 2. http_latency_p95{route!="export"} > 2s         for 10m   -> page
#      "the site is slow"           · runbook: db_ms vs external_ms panel
# 3. celery_queue_oldest_age_s > 300                for 10m   -> page
#      "work is falling behind"     · runbook: worker count, consumer logs
# 4. db_connections_used / db_connections_max > 0.8 for 5m    -> ticket
#      saturation leads: act before the ceiling refuses
# 5. replica_lag_seconds > 30                       for 5m    -> ticket
#      lag is also the data-loss window on failover
#
# Not alerts, deliberately: CPU, memory, single task failures, cache hit rate.
# Five rules, each naming a user-visible symptom and what to do next.


# ---- What Django contributes -------------------------------------------
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,     # keep Django's own loggers working
    "filters": {
        "request_id": {"()": "observability.request_id.RequestIDFilter"},
        # Django ships this: only pass records when DEBUG is False.
        "require_debug_false": {"()": "django.utils.log.RequireDebugFalse"},
    },
    "handlers": {
        "console": {"class": "logging.StreamHandler", "formatter": "json",
                    "filters": ["request_id"]},
    },
    "loggers": {
        # 5XX at ERROR, 4XX at WARNING — already emitted for you.
        "django.request": {"handlers": ["console"], "level": "WARNING", "propagate": False},
        # Suspicious operations and other security errors.
        "django.security": {"handlers": ["console"], "level": "WARNING", "propagate": False},
    },
    "root": {"handlers": ["console"], "level": "INFO"},
}

3–12: Five rules for a whole service. Each names a symptom users would recognise, carries a duration, and has a runbook line — so being woken by one is immediately actionable.
12: Replica lag is included because it is the data-loss window on failover, not only added latency — a ticket rather than a page, but not invisible.
14: The exclusions are part of the design. CPU is a cause, and a single retried task is the retry policy working as intended; alerting on either produces noise that erodes the other five.
20–24: `disable_existing_loggers: False` keeps Django's own loggers alive alongside yours — turning it on silently removes the `django.request` records the first alert depends on.
31–34: Two loggers worth wiring explicitly: `django.request` already logs 5XX at ERROR and 4XX at WARNING, and `django.security` receives suspicious-operation events, so both feed the error-rate signal without any code of your own.

Why this works: A small, symptom-shaped alert set stays trusted, and the Django side of it is configuration rather than instrumentation — the framework already emits the request and security records the alerts count.

Alerting on causes instead of symptoms

Wrong

text

CPU > 80%            · memory > 70%       · disk I/O high
one Celery task failed · a 500 occurred once · deploy finished
  -> dozens of pages a week, most needing no action

Better

text

error_rate > 2% for 5m · p95 > 2s for 10m · oldest queued job > 5m
  -> three pages, each meaning "users are affected, go look"

What you see: On-call rotations become exhausting and people start closing notifications without reading them — so the one alert that mattered is dismissed with the same reflex as the ninety that did not.

Why: There are many causes for any given symptom and only a few symptoms, so cause-based alerting needs far more rules while still missing failures no rule anticipated. High CPU is often healthy; a service can also be completely broken with low CPU. Symptom-based alerting inverts that: a handful of rules covers a wide range of causes, including new ones, and every firing means something a user could notice. The duration clause is the second half — an instantaneous spike is normal behaviour, and only the sustained version is worth a person's attention.

From a page to a root cause, without changing tools blindly

1. The alert fires on a symptom

A metric crossed a threshold and stayed there. It says what is wrong for users, not what caused it, and it links to the dashboard for this service.

2. The dashboard says which layer

Latency, error rate, saturation and queue panels side by side. The point is not detail — it is deciding whether this is database, dependency, queue or capacity.

3. A trace shows where the time went

One sampled request, broken into spans across services. This is the only view that crosses a process boundary and names the slow hop.

4. Logs and the error tracker confirm it

Search by the request id from the trace. The error tracker groups the exception into one issue with a count and a first-seen time, which answers "is this new?".

Alert or dashboard?

Alert or dashboard?
Signal	Alert?	Why
error rate > 2% for 5 min	yes	users are affected now, and someone must look
p95 latency > 2s for 10 min	yes	a symptom, and sustained rather than a spike
oldest queued job > 5 min	yes	invisible everywhere else, and worsens silently
DB connections > 80% of max	yes	saturation leads — there is time to act before it refuses
CPU > 80%	no	a cause, not a symptom; often entirely fine
one task failed and retried	no	retries are the design working; alert on the retry rate
cache hit rate	no	a dashboard number for judging a change

Together

text

alert: celery_queue_oldest_age_s{queue="emails"} > 300 for 10m
runbook: check worker count, then the emails queue consumer log

Remember: Alert on symptoms users feel — error rate, latency, queue age, saturation — never on causes like CPU, because there are many causes and few symptoms. Put a duration on every threshold, and make each alert name what to do. Everything else is a dashboard panel: if the right response to a page is to close it, it should not have been a page. Keep the tool set small and *linked* — alert to dashboard to trace to logs, joined by the request id — and remember Django already emits much of what you need, with `django.request` logging 5XX at ERROR and 4XX at WARNING.