beginnerintermediateadvanced

Django · Section 90

Reliability Patterns

Level: advanced
Read: 34 min
Concepts: 3

The section is a list of twelve mechanisms and one instruction: an external dependency can fail in many ways — timeout, HTTP 500, HTTP 429, connection refused, DNS failure, partial response, malformed response, duplicate success, delayed response — and your system should have deliberate behaviour for each important class. Deliberate is the whole point, because the default behaviour already exists and it is bad: an unhandled exception, a 500 for the entire page, and a worker held until something gives up. Start at the level of one call. A timeout is what converts an unbounded wait into a catchable error, and Requests states its default outright — "By default, requests do not time out unless a timeout value is set explicitly" — so an outbound call without one can hold a gunicorn worker for minutes. Set connect and read separately, then size the retry policy against a budget: three attempts at an eight-second timeout is a twenty-four-second worst case, and the retries have to fit inside the time a user is actually waiting. Retry only what can succeed next time — timeouts, connection errors, 502/503/504, and 429 while obeying its `Retry-After` — because a 400 will fail identically forever. Back off exponentially so a struggling service sees falling pressure, and add jitter so every client that failed at the same instant does not return at the same instant. And retry only what is idempotent: a read timeout means you stopped listening, not that the work did not happen, so an idempotency key is what makes the second attempt free. Move up a level and the question becomes how much of your system one bad dependency may consume. A circuit breaker bounds the time: after N failures it opens and fails instantly, then half-opens to let exactly one trial call decide whether to close — and its state belongs in Redis, because a module-level counter in one of twenty-four workers sees a twenty-fourth of the failures. A bulkhead bounds the resources: separate Celery queues so a render backlog cannot starve password resets, a separate deployment so reporting queries cannot eat checkout's connections. Rate limits bound arrivals in both directions — inbound to protect your own capacity, outbound to stay inside a provider's quota. Finally, decide what the user sees. Degradation means the page works without the part that failed, and a fallback has to be honest: stale-and-labelled beats missing, and missing beats an empty cart that reads as an answer. In background work, dead-letter a message once its attempts are exhausted so a poison payload stops blocking everything behind it — and ship the alert and the replay command with it, or you have only made the loss quieter. Health checks tell the platform what is true: liveness means "restart me" and must do no I/O, readiness means "send me traffic" and may check the database, and failing readiness first is what lets a deploy drain without dropping a request.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

A call without a timeout has no worst case, and the worker is the thing it spends.
Retry the transient, refuse the deterministic, and fit attempts inside a real budget.
Backoff lowers the load; jitter stops the herd arriving together.
Breakers bound time on a bad dependency, bulkheads bound resources, rate limits bound arrivals.
Every fallback is a claim to the user — make it one that is true.

What you will be able to do

Bound every outbound call, and size a retry policy against the request budget
Build a circuit breaker whose state the whole fleet can see
Choose a compartment — queue, pool or deployment — for work that must not starve the rest
Give each failure class a decided behaviour, and each fallback an honest label

One failing dependency, and the four places you get to decide what happens

Outbound call

payment, search, recommendations, webhook

Breaker closed?

open → fail in microseconds, no socket opened

Bounded attempt

timeout=(connect, read) — the worst case is now a number

Success

reset the failure count

Which failure class?

the nine the section names

Transient → retry

backoff + jitter, inside the budget, with an idempotency key

Deterministic → stop

400/404/malformed: the same call will fail identically

Breaker opens

threshold crossed — stop paying the timeout per request

Fallback, labelled

stale, substitute, or hidden — never a silent empty answer

Dead-letter (async only)

with an alert and a replay command

Bounding one call

Timeouts, retries, backoff and jitter — and the idempotency that makes a retry safe.

Timeouts, retries, backoff and jitter

coreadvanced

A **timeout** is the longest you are willing to wait before giving up on a call. A **retry** is trying again after a failure. **Exponential backoff** doubles the wait between attempts, so a struggling service gets quieter traffic instead of more. **Jitter** adds randomness to that wait, so a thousand clients do not all come back at the same instant. Retrying is only safe when the call is idempotent — running it twice has the same effect as running it once.

Think of it as

Start from the fact that every remote call has an unbounded worst case unless you bound it. Requests states it plainly: "By default, requests do not time out unless a timeout value is set explicitly. Without a timeout, your code may hang for minutes or more." A gunicorn worker stuck in that call is a worker serving nobody, and the failure spreads outward — the worker pool fills, the queue behind it fills, and a slow dependency you do not own takes your site down. So the timeout is not a nicety, it is the thing that converts *their* outage into *your* handled error. Set two numbers, not one: the connect timeout bounds establishing the TCP connection, the read timeout bounds waiting for bytes after the request is sent, and `timeout=(3.05, 27)` sets them separately. Next comes the budget, and this is where most retry code goes wrong. The user is waiting for one HTTP response, so the total time you may spend is fixed — say 10 seconds. Three attempts at a 10-second timeout is a 30-second worst case, which means the retries do not make the call more reliable, they make the request time out somewhere further up. Pick the per-attempt timeout so that attempts × timeout + waits fits inside the budget you actually have. Then decide what is worth retrying at all. A timeout, a connection refused, a 502/503/504 — those are transient and may succeed on the next attempt. A 400, a 401, a 404, a validation error — those will fail identically forever, and retrying them only spends your budget. A 429 is special: it is the server telling you the rate is too high, and it often carries `Retry-After`, which you should obey rather than compute. Backoff exists because a failing service is usually failing *because* of load, and a fixed 100 ms retry loop from every client is a denial-of-service attack you wrote yourself. Doubling — 1s, 2s, 4s, 8s — drops the pressure fast. Jitter exists because backoff alone still synchronises: every client that saw the outage at the same moment retries at the same moment, so the recovering service gets a wall of traffic, falls over again, and the herd re-synchronises. Randomising the sleep spreads the same number of attempts across the window. The last piece is the one that is not about waiting at all. A retry can duplicate work, because a timeout does not tell you whether the other side did the thing — it tells you that you did not hear back. If the call charges a card, the safe version sends an idempotency key so the second attempt returns the first attempt's result instead of charging twice.

python

requests.post(url, json=payload, timeout=(3.05, 10))  # (connect, read)

What we're doing: Call a payment provider with a bounded budget, a retry policy that knows what is worth retrying, and a duplicate-safe key.

billing/gateway.pypython

import random
import time

import requests

RETRYABLE_STATUS = {429, 500, 502, 503, 504}
MAX_ATTEMPTS = 3
CONNECT_TIMEOUT = 3.05     # slightly over a multiple of 3 — the TCP
READ_TIMEOUT = 8.0         # retransmission window is 3 seconds


def charge(*, order_id, amount_cents, idempotency_key):
    # 3 attempts x 8s read + waits stays inside a ~30s request budget.
    # Raising MAX_ATTEMPTS without lowering READ_TIMEOUT breaks that.
    last_error = None

    for attempt in range(MAX_ATTEMPTS):
        try:
            response = requests.post(
                "https://api.provider.example/v1/charges",
                json={"order_id": order_id, "amount": amount_cents},
                # The provider returns the FIRST result for a repeated key,
                # so a retry after a timeout cannot charge twice.
                headers={"Idempotency-Key": idempotency_key},
                timeout=(CONNECT_TIMEOUT, READ_TIMEOUT),
            )
        except (requests.ConnectionError, requests.Timeout) as exc:
            last_error = exc
        else:
            if response.status_code not in RETRYABLE_STATUS:
                # Includes 400 and 422: a bad payload fails identically
                # forever, so returning now saves the rest of the budget.
                return response
            last_error = f"HTTP {response.status_code}"
            if response.status_code == 429 and "Retry-After" in response.headers:
                time.sleep(float(response.headers["Retry-After"]))
                continue

        if attempt < MAX_ATTEMPTS - 1:
            # Full jitter. Without random(), every client that saw this
            # outage retries at the same three instants.
            time.sleep(random.uniform(0, min(10.0, 2**attempt)))

    raise GatewayUnavailable(f"charge failed after {MAX_ATTEMPTS} attempts: {last_error}")

8–9: Two separate numbers. The connect timeout bounds reaching the host; the read timeout bounds waiting for a response after the request is sent. Requests applies a single value to both, which usually means one of them is wrong.
13–14: The budget is stated as a comment because it is the constraint the other numbers must satisfy. Attempts and per-attempt timeout multiply — three attempts at 30 seconds is a 90-second worst case no user is waiting for.
22–24: The key makes the retry safe. Without it a read timeout leaves you unable to tell "the charge did not happen" from "the charge happened and the reply was lost", and the only safe choice would be not to retry at all.
30–33: Non-retryable statuses return immediately. Retrying a 400 spends the budget on a request whose outcome cannot change.
35–37: `Retry-After` is the server's own number. Obeying it beats any backoff you compute, because the server knows when the limit resets and you do not.
39–42: Full jitter: sleep uniformly between zero and the exponential backoff for this attempt. The mean wait is halved and the arrivals stop lining up.

Why this works: The worst case is bounded and known, only transient failures consume attempts, a struggling provider sees decreasing and spread-out load, and a duplicate attempt cannot charge a customer twice.

Retrying without an idempotency key after a read timeout

Wrong

python

for attempt in range(3):
    try:
        return requests.post(CHARGE_URL, json=payload, timeout=10)
    except requests.Timeout:
        continue  # the first charge may have SUCCEEDED

Better

python

headers = {"Idempotency-Key": idempotency_key}  # same key on every attempt
return requests.post(CHARGE_URL, json=payload, headers=headers, timeout=(3.05, 10))

What you see: A small, steady trickle of customers charged two or three times, always during a period when the provider was slow rather than down — and no error in your logs, because every attempt that mattered succeeded.

Why: A read timeout means you stopped waiting. It does not mean the other side stopped working: the request may have arrived, been processed and committed, with only the response lost. So the retry is a second, independent charge. An idempotency key closes the gap by moving deduplication to the side that knows — the provider stores the key with the first result and returns that same result for any later request carrying it. The key must be generated once, per logical operation, and reused across attempts; generating it inside the loop gives every attempt a new key and restores the original bug.

Nine retries, two schedules — why jitter is not a detail

Backoff decides how much total load a recovering service sees; jitter decides whether it arrives as three spikes or as a spread. The attempt count is identical in both panels.

What each failure class deserves

What each failure class deserves
Failure	Retry?	Why
connection refused / DNS failure	yes, with backoff	the request never reached the application
read timeout	only if idempotent	the work may already have happened
HTTP 500 / 502 / 503 / 504	yes, with backoff	transient by definition of the status code
HTTP 429	yes — obey `Retry-After`	the server is telling you the rate, do not guess
HTTP 400 / 422 (bad payload)	no	the same bytes will fail identically forever
HTTP 401 / 403 / 404	no	retrying spends the budget and changes nothing

Together

python

RETRYABLE_STATUS = {429, 500, 502, 503, 504}
RETRYABLE_ERRORS = (requests.ConnectionError, requests.Timeout)

The four waiting strategies, on the same failing call

The four waiting strategies, on the same failing call
Strategy	Waits before attempts 2·3·4	What it does to a struggling service
immediate retry	0s · 0s · 0s	triples the load on the thing that is already failing
fixed delay	1s · 1s · 1s	steady pressure; the herd stays synchronised
exponential backoff	1s · 2s · 4s	pressure falls fast, but every client still lands together
backoff + full jitter	rand(0,1) · rand(0,2) · rand(0,4)	same number of attempts, spread across the window

Together

python

import random

def sleep_for(attempt, *, base=1.0, cap=30.0):
    """Full jitter: pick uniformly from [0, the backoff for this attempt]."""
    return random.uniform(0, min(cap, base * 2 ** attempt))

Remember: Every outbound call gets a timeout, because `requests` has none by default and a hung call holds a worker forever. Set connect and read separately, and size attempts × timeout to fit the request budget you actually have. Retry only transient failures — timeouts, connection errors, 502/503/504, and 429 with its `Retry-After` — never a 400 or a 404. Back off exponentially so a struggling service sees less traffic, and add jitter so every client does not come back at the same instant. And retry only what is idempotent: a read timeout does not tell you whether the work happened, so carry an idempotency key generated once and reused on every attempt.

Containing one dependency

How much time, how many resources and what arrival rate a single dependency may consume.

Circuit breakers, bulkheads and rate limits

coreadvanced

A **circuit breaker** watches a dependency and, after enough failures, stops calling it for a while — failing instantly instead of waiting for a timeout every time. A **bulkhead** caps how much of a shared resource one kind of work may use, so a flood in one place cannot drain everything. **Rate limiting** caps how many calls happen per unit of time, either to protect you from a caller or to keep you inside a provider's quota.

Think of it as

Timeouts and retries make one call behave; these three make one *dependency* behave. The circuit breaker starts from a problem timeouts create: if a provider is down and your timeout is 8 seconds, every request pays 8 seconds to learn what the previous hundred already proved. The breaker keeps a small piece of state — recent failure count — and once it crosses a threshold it opens: calls fail immediately, with no socket, no wait and no worker held. After a cooling-off period it goes half-open and lets a single trial call through. That call decides: success closes the breaker, failure opens it again for another period. The half-open state is the part people leave out, and without it you either stay open forever or slam the recovering service with the full backlog the moment the timer expires. Where the breaker limits *time spent* on a bad dependency, the bulkhead limits *resources committed* to it. The name is from ship compartments: flood one and the ship floats, because the flooding cannot reach the rest. In a Django deployment the compartments are real and concrete. Separate Celery queues mean a backlog of slow PDF renders cannot starve password-reset emails, because they have their own workers. A separate connection pool, or a separate deployment, means the reporting endpoints that hold long queries cannot consume the connections the checkout path needs. A cap on concurrent outbound calls to one provider means that provider can never occupy more than, say, four of your eight workers — a self-imposed limit that guarantees the other four keep serving. Rate limiting is the same containment idea applied to arrival rate, and it points in both directions. Inbound, it protects you: DRF throttling caps what one client can do per minute, so a broken script or an abusive caller cannot consume the capacity everyone shares. Outbound, it protects the relationship: a provider that allows 100 requests per minute will return 429 if you exceed it, and a token bucket on your side is how you stay under the line rather than discovering it in production. The unifying idea is that all three answer one question — what is the *most* damage this dependency is allowed to do? — and they answer it with a number chosen in advance rather than with whatever the incident happens to produce.

python

with breaker("payments"):        # raises BreakerOpen instead of waiting
    response = gateway.charge(...)

What we're doing: Stop paying an 8-second timeout for every request while a provider is down, using state every process can see.

reliability/breaker.pypython

from contextlib import contextmanager

from django.core.cache import cache


class BreakerOpen(Exception):
    """The dependency is known to be failing; we did not call it."""


@contextmanager
def breaker(name, *, threshold=5, reset_after=30):
    # Redis, not a module-level dict. With 24 gunicorn workers a local
    # counter sees 1/24 of the failures and opens 24 times too late.
    fail_key = f"breaker:{name}:failures"
    open_key = f"breaker:{name}:open"
    trial_key = f"breaker:{name}:trial"

    if cache.get(open_key):
        # OPEN. The cooling-off period has not expired: fail now, having
        # opened no socket and held no worker.
        raise BreakerOpen(name)

    # HALF-OPEN: the open key has expired, so exactly one caller wins the
    # add() race and gets to make the trial call. Everyone else still
    # fails fast, which is what stops a thundering herd on recovery.
    is_trial = cache.get(fail_key, 0) >= threshold
    if is_trial and not cache.add(trial_key, "1", timeout=reset_after):
        raise BreakerOpen(name)

    try:
        yield
    except Exception:
        # incr() is atomic; get-then-set would lose failures under load.
        if cache.add(fail_key, 1, timeout=reset_after * 4) is False:
            failures = cache.incr(fail_key)
        else:
            failures = 1
        if failures >= threshold:
            cache.set(open_key, "1", timeout=reset_after)
        raise
    else:
        # Success closes the breaker and clears the history in one step.
        cache.delete_many([fail_key, open_key, trial_key])

11–13: The state lives in the shared cache because the decision is fleet-wide. A per-process counter is the single most common way a breaker silently does nothing: each of 24 workers has to fail five times independently before any of them opens.
18–21: This is the whole payoff. While the breaker is open a request costs microseconds instead of the full read timeout, so the worker pool stays free and the rest of the site keeps serving.
23–28: Half-open, implemented as a race exactly one caller wins. Letting every caller through the moment the timer expires would hand a recovering provider the entire backlog at once.
33–39: Failures are counted with `incr`, which is atomic. A read-modify-write would lose counts precisely when traffic is highest, which is when the breaker matters.
41–43: One success clears the failure history. Decaying the count instead would keep a breaker half-armed for a service that has already recovered.

Why this works: While the provider is down, requests fail in microseconds rather than seconds, no worker is held, and recovery is probed by one call at a time instead of by the entire backlog.

Keeping breaker state in a module-level variable

Wrong

python

_failures = 0          # per PROCESS, and gunicorn runs 24 of them

def call_provider():
    global _failures
    ...

Better

python

cache.incr(f"breaker:{name}:failures")   # one counter the whole fleet shares

What you see: The breaker "works" in a single-process dev server and never trips in production; during an outage every worker keeps paying the full timeout, and restarting the fleet resets whatever state had accumulated.

Why: Gunicorn workers are separate OS processes with separate memory, so a module-level counter is one counter per worker. With 24 workers and a threshold of 5, the fleet must absorb 120 failures before every worker has opened — and each worker's count vanishes when it is recycled by `--max-requests`. The state has to live where every process can see it, which in a Django deployment is the cache backend you already run for sessions and throttling.

A circuit breaker, including the state everyone forgets

closed — calls go out, failures counted

start

open — fail instantly, nothing is called

half-open — exactly one trial call

The three states, and what each does to a call

The three states, and what each does to a call
State	The call	Leaves this state when
closed (normal)	goes out; failures are counted	failures in the window cross the threshold
open	raises immediately — no socket, no wait	the cooling-off period expires
half-open	one trial call is allowed through	it succeeds → closed · it fails → open again

Together

python

# Thresholds are policy, not defaults to copy:
FAILURE_THRESHOLD = 5     # consecutive failures before opening
RESET_AFTER = 30          # seconds open before one trial call

Three compartments a Django deployment already has

Three compartments a Django deployment already has
Bulkhead	What it contains	What leaks without it
separate Celery queues	slow tasks (PDF, export, video)	a render backlog delays password-reset emails
separate web deployment for reports	long queries and big payloads	report traffic eats the connections checkout needs
semaphore on outbound calls	one provider's share of the worker pool	a slow provider occupies every worker at once

Together

python

# Two queues, two worker pools — one cannot starve the other.
send_password_reset.apply_async(queue="critical")
render_invoice_pdf.apply_async(queue="slow")

Remember: A circuit breaker stops you paying a timeout per request for a dependency already known to be down: after N failures it opens and fails instantly, then half-opens and lets exactly one trial call decide. Keep its state in Redis — a per-process counter sees 1/N of the traffic and never trips. A bulkhead caps what one dependency may consume: separate Celery queues, separate deployments, a cap on concurrent outbound calls. Rate limits cap arrival rate in both directions — inbound to protect your capacity, outbound to stay inside a provider's quota. And only break a circuit you have a fallback for; otherwise you have made the same failure arrive faster.

Deciding what the user and the platform see

Degradation, honest fallbacks, dead-letter queues, and the two health checks.

Degrade, fall back, dead-letter, health-check

coreadvanced

**Graceful degradation** means the page still works when a part of it does not — the recommendations panel disappears, the order still goes through. A **fallback** is the specific answer you give instead: stale cache, a default, an empty list. A **dead-letter queue** holds messages that failed too many times, so they stop blocking the queue and can be looked at later. A **health check** is the endpoint your load balancer calls to decide whether this instance should receive traffic.

Think of it as

The roadmap lists nine ways a dependency fails — timeout, 500, 429, connection refused, DNS failure, partial response, malformed response, duplicate success, delayed response — and asks for deliberate behaviour for each important class. Deliberate is the operative word: the default behaviour is an unhandled exception that becomes a 500 for the whole page, and that is a decision made by omission. Degradation is the practice of deciding what the page is *without* each dependency. Ask it feature by feature. If recommendations are unavailable, show this week's popular items; if the avatar service is down, show initials; if the search cluster is down, fall back to a database `icontains` query and say results may be incomplete. What you must not do is fall back silently in a way that looks like an answer. Returning an empty cart because the cart service timed out is worse than an error, because the user believes it. So the rule is that a fallback is honest: it is either indistinguishable in meaning from the real answer, or it is labelled. Stale-but-marked beats missing, and missing beats wrong. The same reasoning applies one layer down, in background work. A message that fails deterministically — malformed payload, a row that no longer exists — will fail every time it is retried, and while it is being retried it can hold up everything behind it. The dead-letter queue is where such a message goes after its attempts are exhausted: out of the way, retained, and countable. A dead-letter queue with no alert and no replay path is not a safety net, it is a slower way to lose data; the useful version has a metric you can alarm on and a management command that re-drives a message once the bug is fixed. Health checks close the loop by telling the infrastructure what is true right now, and they come in two kinds that get conflated at some cost. Liveness answers "is this process wedged?" and should fail only when a restart is the fix — so it must not check the database, or a database blip will restart every container you own. Readiness answers "should this instance receive traffic?" and may check the things a request needs. Readiness is also what makes zero-downtime deploys work: on shutdown you flip readiness to failing first, keep serving in-flight requests, and let the load balancer stop sending new ones before the process exits.

python

except (BreakerOpen, requests.RequestException):
    return fallback_value, {"degraded": True, "reason": "recommendations"}

What we're doing: Serve a product page that keeps working when the recommender is down, and expose the two health checks a rolling deploy needs.

catalog/views.py + ops/health.pypython

# catalog/views.py
def product_detail(request, sku):
    product = get_object_or_404(Product, sku=sku)   # no fallback exists
    degraded = []

    try:
        with breaker("recommendations"):
            picks = recommender.for_user(request.user, sku=sku)
    except (BreakerOpen, requests.RequestException):
        # Substitute, not silence. The page still recommends something,
        # and the template can say these are popular rather than personal.
        picks = popular_this_week()
        degraded.append("recommendations")
        logger.warning("degraded", extra={"feature": "recommendations", "sku": sku})

    return render(request, "catalog/detail.html", {
        "product": product,
        "picks": picks,
        "degraded": degraded,      # the template LABELS what is degraded
    })


# ops/health.py
def liveness(request):
    """Can this process still answer? No I/O — a database blip must not
    restart every container in the fleet."""
    return HttpResponse("ok", content_type="text/plain")


def readiness(request):
    """Should the load balancer send this instance traffic?"""
    if cache.get("shutting_down"):
        # Set by the SIGTERM handler. Failing readiness BEFORE we stop
        # accepting connections is what makes the deploy lose no requests.
        return HttpResponse("draining", status=503)

    try:
        connection.ensure_connection()
    except OperationalError:
        return HttpResponse("db unavailable", status=503)

    return HttpResponse("ready", content_type="text/plain")

3: The product itself has no fallback — without it there is no page. Naming that explicitly is part of the exercise: degradation applies to the dependencies that have an alternative, not to every call.
10–14: The substitute answer plus a record of which feature degraded. The log line is what turns a quiet fallback into something you can alert on and count.
19: Passing `degraded` into the template is what keeps the fallback honest. A user who sees "popular this week" instead of "picked for you" has not been misled.
24–27: Liveness does no I/O at all. A liveness probe that touches the database turns a five-second database hiccup into a fleet-wide restart, which is strictly worse than the hiccup.
32–35: The drain flag. On SIGTERM you flip this first, keep serving in-flight requests for the load balancer's next check interval, and only then let the process exit.
37–40: `ensure_connection()` is Django's own way to prove the database is reachable without inventing a query. A failure here removes this instance from the pool rather than restarting it.

Why this works: A recommender outage costs a panel, not a page; the health endpoints answer two different questions with two different blast radii; and the drain flag makes a rolling deploy invisible to users.

A silent fallback that looks like a real answer

Wrong

python

try:
    items = cart_service.get(user)
except Exception:
    items = []          # renders as "your cart is empty"

Better

python

try:
    items = cart_service.get(user)
except Exception:
    raise CartUnavailable   # the page says "we can't load your cart right now"

What you see: Support tickets saying items vanished from carts. Nothing is in the error logs, because the code handled the exception, and the checkout funnel shows an unexplained drop that no deploy accounts for.

Why: An empty list and "we could not reach the cart service" are different facts, and rendering the first when the second is true tells the user something false. Users act on it — they re-add items, or they leave. The test for any fallback is whether a reader can distinguish it from the real answer: `[]` for a cart fails that test, while "popular this week" under a different heading passes it. Where no honest substitute exists, an error is the correct output, and it should be an error you logged.

The degradation ladder — each rung is a decision you make in advance

Full service

every dependency healthy: live recommendations, live search, live avatars

Stale but labelled

serve the cached answer and say when it was computed — "prices as of 09:41"

Substitute answer

popular items instead of personalised ones; initials instead of an avatar

Feature hidden

the panel is not rendered at all — nothing on the page is wrong, only less

Read-only

browsing works, writes are refused with a clear message and a retry time

Error page

the last rung, reached only when the request cannot be answered at all

The roadmap's failure classes, and a deliberate behaviour for each

The roadmap's failure classes, and a deliberate behaviour for each
Failure	What it means	Deliberate behaviour
Timeout	you stopped waiting; the work may have happened	retry only with an idempotency key, then fall back
HTTP 500	their bug, probably transient	retry with backoff, then fall back
HTTP 429	you are over their limit	obey `Retry-After`; slow the caller, do not hammer
Connection refused / DNS failure	nothing was reached	safe to retry — no side effect can have occurred
Partial or malformed response	you got bytes you cannot trust	do not retry — validate, log the body, dead-letter
Duplicate success	you did the work twice	deduplicate on your side; this is what idempotency is for
Delayed response	it arrived after you gave up	ignore it — the request it belonged to is gone

Together

python

try:
    picks = recommender.for_user(user, timeout=(2, 3))
except (BreakerOpen, requests.RequestException):
    picks, degraded = popular_this_week(), True   # labelled, not silent

Liveness and readiness are different questions

Liveness and readiness are different questions
Dimension	Liveness `/healthz`	Readiness `/readyz`
answers	is this process wedged?	should this instance get traffic?
on failure the platform	restarts the container	removes it from the pool
may check the database	no — a DB blip would restart the fleet	yes — a request needs it
during shutdown	stays passing until the process exits	fails first, so the LB drains you
cost budget	microseconds, no I/O	one cheap query, cached for a few seconds

Together

python

path("healthz", lambda r: HttpResponse("ok")),   # no I/O, no auth, no DB
path("readyz", readiness_view),                  # checks DB + cache

Remember: Decide in advance what each page is without each dependency, because the default decision is a 500. Fall back honestly — stale-and-labelled beats missing, and missing beats a wrong answer like an empty cart. Dead-letter a message once its attempts are exhausted so it stops blocking the queue, and ship the alert and the replay command with it, or you have only made the loss quieter. Keep liveness and readiness separate: liveness does no I/O and means "restart me", readiness may check the database and means "send me traffic" — and failing readiness first is what lets a deploy drain without dropping a request.