beginnerintermediateadvanced

Django · Reliability Patterns

Reliability Patterns

Decide what happens when a dependency fails, before it does

No timeout means no bound`requests` waits forever by default — and holds a worker while it does
Retry only what is transienttimeouts, connection errors, 502/503/504, 429 — never a 400
Jitter, not just backoffthe herd that failed together comes back together without it
A breaker needs shared statea per-process counter sees 1/24 of the failures and never trips
A fallback must be honestan empty cart is worse than an error, because users believe it

1.Timeouts, retries, backoff and jitter

A **timeout** is the longest you are willing to wait before giving up on a call. A **retry** is trying again after a failure. **Exponential backoff** doubles the wait between attempts, so a struggling service gets quieter traffic instead of more. **Jitter** adds randomness to that wait, so a thousand clients do not all come back at the same instant. Retrying is only safe when the call is idempotent — running it twice has the same effect as running it once.

Key features

`requests` has no default timeout — an unbounded call ties up a worker forever
Two timeouts, not one: connect (reach the server) and read (wait for bytes)
A retry budget is total: attempts × per-attempt timeout must fit the request budget
Retry timeouts, connection errors, 502/503/504 — never 400/401/403/404
Backoff drops load on a struggling service; jitter stops the herd re-synchronising
Retry only what is idempotent, or carry an idempotency key so the duplicate is free

What each failure class deserves

What each failure class deserves
Failure	Retry?	Why
connection refused / DNS failure	yes, with backoff	the request never reached the application
read timeout	only if idempotent	the work may already have happened
HTTP 500 / 502 / 503 / 504	yes, with backoff	transient by definition of the status code
HTTP 429	yes — obey `Retry-After`	the server is telling you the rate, do not guess
HTTP 400 / 422 (bad payload)	no	the same bytes will fail identically forever
HTTP 401 / 403 / 404	no	retrying spends the budget and changes nothing

The four waiting strategies, on the same failing call

The four waiting strategies, on the same failing call
Strategy	Waits before attempts 2·3·4	What it does to a struggling service
immediate retry	0s · 0s · 0s	triples the load on the thing that is already failing
fixed delay	1s · 1s · 1s	steady pressure; the herd stays synchronised
exponential backoff	1s · 2s · 4s	pressure falls fast, but every client still lands together
backoff + full jitter	rand(0,1) · rand(0,2) · rand(0,4)	same number of attempts, spread across the window

Nine retries, two schedules — why jitter is not a detail

Backoff decides how much total load a recovering service sees; jitter decides whether it arrives as three spikes or as a spread. The attempt count is identical in both panels.

Command

python

requests.post(url, json=payload, timeout=(3.05, 10))  # (connect, read)

Every outbound call gets a timeout, because `requests` has none by default and a hung call holds a worker forever. Set connect and read separately, and size attempts × timeout to fit the request budget you actually have. Retry only transient failures — timeouts, connection errors, 502/503/504, and 429 with its `Retry-After` — never a 400 or a 404. Back off exponentially so a struggling service sees less traffic, and add jitter so every client does not come back at the same instant. And retry only what is idempotent: a read timeout does not tell you whether the work happened, so carry an idempotency key generated once and reused on every attempt.

2.Circuit breakers, bulkheads and rate limits

A **circuit breaker** watches a dependency and, after enough failures, stops calling it for a while — failing instantly instead of waiting for a timeout every time. A **bulkhead** caps how much of a shared resource one kind of work may use, so a flood in one place cannot drain everything. **Rate limiting** caps how many calls happen per unit of time, either to protect you from a caller or to keep you inside a provider's quota.

Key features

Open breaker = fail immediately, no socket opened, no worker held for the timeout
Half-open is mandatory: one trial call decides close or re-open
A breaker needs shared state (Redis) — per-process counters see 1/N of the traffic
Bulkhead = a cap on resources one dependency may consume: workers, connections, queues
Inbound rate limits protect you; outbound rate limits keep you inside a provider quota
Never wrap a breaker around a dependency you have no fallback for — you just fail faster

The three states, and what each does to a call

The three states, and what each does to a call
State	The call	Leaves this state when
closed (normal)	goes out; failures are counted	failures in the window cross the threshold
open	raises immediately — no socket, no wait	the cooling-off period expires
half-open	one trial call is allowed through	it succeeds → closed · it fails → open again

Three compartments a Django deployment already has

Three compartments a Django deployment already has
Bulkhead	What it contains	What leaks without it
separate Celery queues	slow tasks (PDF, export, video)	a render backlog delays password-reset emails
separate web deployment for reports	long queries and big payloads	report traffic eats the connections checkout needs
semaphore on outbound calls	one provider's share of the worker pool	a slow provider occupies every worker at once

A circuit breaker, including the state everyone forgets

5 failures inthe windowafter 30scooling offtrial callsucceedstrial call fails— wait againsuccess resetsthe counter

closed — calls go out, failures counted

start

open — fail instantly, nothing is called

half-open — exactly one trial call

Command

python

with breaker("payments"):        # raises BreakerOpen instead of waiting
    response = gateway.charge(...)

A circuit breaker stops you paying a timeout per request for a dependency already known to be down: after N failures it opens and fails instantly, then half-opens and lets exactly one trial call decide. Keep its state in Redis — a per-process counter sees 1/N of the traffic and never trips. A bulkhead caps what one dependency may consume: separate Celery queues, separate deployments, a cap on concurrent outbound calls. Rate limits cap arrival rate in both directions — inbound to protect your capacity, outbound to stay inside a provider's quota. And only break a circuit you have a fallback for; otherwise you have made the same failure arrive faster.

3.Degrade, fall back, dead-letter, health-check

**Graceful degradation** means the page still works when a part of it does not — the recommendations panel disappears, the order still goes through. A **fallback** is the specific answer you give instead: stale cache, a default, an empty list. A **dead-letter queue** holds messages that failed too many times, so they stop blocking the queue and can be looked at later. A **health check** is the endpoint your load balancer calls to decide whether this instance should receive traffic.

Key features

Decide the page's behaviour for each failure class — the default is a 500 for everything
A fallback must be honest: stale-but-labelled beats missing, missing beats wrong
Dead-letter after N attempts so a poison message stops blocking the queue behind it
A DLQ needs an alert and a replay command, or it is silent data loss
Liveness = restart me. Readiness = send me traffic. Liveness must not check the database
Failing readiness first, then draining, is what makes a rolling deploy lose no requests

The roadmap's failure classes, and a deliberate behaviour for each

The roadmap's failure classes, and a deliberate behaviour for each
Failure	What it means	Deliberate behaviour
Timeout	you stopped waiting; the work may have happened	retry only with an idempotency key, then fall back
HTTP 500	their bug, probably transient	retry with backoff, then fall back
HTTP 429	you are over their limit	obey `Retry-After`; slow the caller, do not hammer
Connection refused / DNS failure	nothing was reached	safe to retry — no side effect can have occurred
Partial or malformed response	you got bytes you cannot trust	do not retry — validate, log the body, dead-letter
Duplicate success	you did the work twice	deduplicate on your side; this is what idempotency is for
Delayed response	it arrived after you gave up	ignore it — the request it belonged to is gone

Liveness and readiness are different questions

Liveness and readiness are different questions
Dimension	Liveness `/healthz`	Readiness `/readyz`
answers	is this process wedged?	should this instance get traffic?
on failure the platform	restarts the container	removes it from the pool
may check the database	no — a DB blip would restart the fleet	yes — a request needs it
during shutdown	stays passing until the process exits	fails first, so the LB drains you
cost budget	microseconds, no I/O	one cheap query, cached for a few seconds

The degradation ladder — each rung is a decision you make in advance

Full service

every dependency healthy: live recommendations, live search, live avatars

Stale but labelled

serve the cached answer and say when it was computed — "prices as of 09:41"

Substitute answer

popular items instead of personalised ones; initials instead of an avatar

Feature hidden

the panel is not rendered at all — nothing on the page is wrong, only less

Read-only

browsing works, writes are refused with a clear message and a retry time

Error page

the last rung, reached only when the request cannot be answered at all

Command

python

except (BreakerOpen, requests.RequestException):
    return fallback_value, {"degraded": True, "reason": "recommendations"}

Decide in advance what each page is without each dependency, because the default decision is a 500. Fall back honestly — stale-and-labelled beats missing, and missing beats a wrong answer like an empty cart. Dead-letter a message once its attempts are exhausted so it stops blocking the queue, and ship the alert and the replay command with it, or you have only made the loss quieter. Keep liveness and readiness separate: liveness does no I/O and means "restart me", readiness may check the database and means "send me traffic" — and failing readiness first is what lets a deploy drain without dropping a request.

Spotted a mistake on this page? Tell us.

Read Sheet