beginnerintermediateadvanced

System Design · Retry Strategy

Retry Strategy

Why uncontrolled retries can amplify an outage, and the controls that make retries safe

Retry stormsuncontrolled retries add load exactly when it hurts most
Backoff + jittergrow the wait, then randomize it to avoid synchronized bursts
Transient vs permanent503/429 worth retrying; 400/404 never will succeed
Max attempts + DLQbound the retry loop, don't silently drop what exhausts it

1.Retries can amplify outages if uncontrolled

A retry is meant to paper over a brief, one-off failure — but when a downstream service is already struggling under load, every client retrying its failed request adds MORE load onto the exact service that is failing, at the exact moment it can least handle it. Uncontrolled retries — no cap on attempts, no backoff, every client retrying instantly and in lockstep — can turn a small, recoverable blip into a sustained, self-sustaining outage, because the retry traffic itself becomes the dominant source of load.

Key features

A retry storm happens when a large number of clients all retry a failing request at roughly the same time, multiplying load on the very service that is already struggling
Retry amplification compounds through a call chain — if service A calls B calls C, and each layer retries independently, one failure at C can turn into a multiplicative fan-out of requests hitting C
A service returning errors under overload, combined with clients retrying aggressively, can prevent that service from ever recovering — the failure becomes self-sustaining instead of transient
The danger is specifically "uncontrolled" retries — no attempt cap, no backoff, no jitter, and no awareness of how many layers upstream are also retrying the same logical request

Retry policy choices and their effect on downstream load

Retry policy choices and their effect on downstream load
Policy	Effect on a struggling downstream service
No retries	Failures are visible immediately but no amplification — safest for downstream, worst for client-perceived reliability on brief blips
Immediate retry, no cap	Instantly re-adds full load; if the cause was overload, this piles on more of exactly what caused the failure
Retry with exponential backoff	Load from each individual client fades over time, giving the downstream service room to recover
Retry with backoff + jitter	Same fading load, but spread out in time across clients instead of arriving in synchronized bursts
Retry with backoff + jitter + max attempts	Bounded worst case — a client eventually gives up and surfaces the failure instead of retrying forever

A 2-second blip amplifies to 27x, retried at every layer

up to 3xup to 9xup to 27x

Frontend

retries ×3

Service A

retries ×3 → up to 9x

Service B

retries ×3 → up to 27x

Database

2s blip, now overloaded

Command

text

retry storm = many clients x retrying together x
               against an already-overloaded service
             = load INCREASES exactly when capacity
               is already the constraint

Retries fix brief, one-off failures — but uncontrolled retries (no cap, no backoff, every layer retrying independently) add load onto a service at exactly the moment it is least able to handle more, turning a small blip into a sustained, self-inflicted outage.

2.Timeouts, exponential backoff and jitter

Three techniques work together to make retries safe. A timeout bounds how long a client waits before deciding a request has failed, so a single slow call cannot hang a caller indefinitely. Exponential backoff waits progressively longer between each retry attempt (e.g. 1s, 2s, 4s, 8s) instead of retrying instantly, giving a struggling downstream service increasing room to recover. Jitter adds randomness to that wait time so that many clients retrying after the same failure do not all retry at the exact same moment — without jitter, backoff alone can still produce synchronized bursts of retries arriving together.

Key features

A timeout should be set per call based on that call's expected latency — a timeout that is too long delays failure detection; one that is too short causes retries of requests that would have succeeded
Exponential backoff multiplies the wait time by a fixed factor (commonly 2x) after each failed attempt: e.g. 1s, 2s, 4s, 8s, 16s — the wait grows without bound unless capped
A cap (maximum backoff) is standard practice — otherwise a long string of failures produces waits so long that a request effectively never retries again in practice
Jitter adds randomness to the backoff wait so retries from different clients spread out over time instead of clustering — full jitter picks a random wait anywhere between 0 and the computed backoff value; equal jitter keeps half the computed value fixed and randomizes only the other half
Backoff and jitter solve different problems: backoff reduces the total volume of retry traffic over time; jitter spreads that traffic out so it does not arrive in synchronized bursts even when volume is otherwise the same

Backoff strategies, in order of increasing protection against synchronized retries

Backoff strategies, in order of increasing protection against synchronized retries
Strategy	Wait calculation	Weakness
No backoff	Fixed, immediate retry every time	Every client retries at the same rate — easiest to cause a retry storm
Exponential backoff, no jitter	wait = base * 2^attempt, capped	Clients that failed at the same moment retry at the same later moment too — still synchronized
Exponential backoff, full jitter	wait = random(0, base * 2^attempt)	Retry timing is spread across the whole window — most effective at avoiding synchronized bursts
Exponential backoff, equal jitter	wait = (base * 2^attempt / 2) + random(0, base * 2^attempt / 2)	Still spreads retries, with a higher guaranteed minimum wait than full jitter

4 clients, all failed at t=0 — retry timing

No jitter

All 4 clients wait exactly 2s
All 4 retry together at t=2s
Every retry arrives as one synchronized burst

Full jitter

Waits randomized: 0.3s, 1.7s, 0.9s, 1.4s
Retries spread across the ~2s window
No synchronized spike, smooth traffic instead

Command

text

# exponential backoff with full jitter (common pattern)
backoff = min(cap, base * 2 ** attempt)
wait    = random_between(0, backoff)

Timeout bounds how long one attempt waits; exponential backoff grows the wait between attempts (capped, so it doesn't grow forever); jitter randomizes that wait so many clients retrying after the same failure don't retry in a synchronized burst. All three work together — none of them alone is a complete retry strategy.

3.Differentiate transient errors from permanent errors

Not every failure is worth retrying. A transient error is one that has a real chance of succeeding on a later attempt — a brief network blip, a momentary connection-pool exhaustion, a downstream service that is temporarily overloaded (often signaled by a 503 or 429 status). A permanent error will fail again in exactly the same way no matter how many times it is retried — malformed input, a missing resource (404), an authorization failure (401/403), or a request that violates a business rule. Retrying a permanent error wastes attempts, adds latency, and adds pointless load to the downstream service for zero chance of success.

Key features

Transient error signals: HTTP 503 (Service Unavailable), 429 (Too Many Requests), connection timeouts, connection resets — these describe a temporary condition that can plausibly resolve itself
Permanent error signals: HTTP 400 (Bad Request), 401/403 (auth), 404 (Not Found), 422 (Unprocessable Entity) — these describe a condition that retrying the identical request cannot fix
A 429 response often comes with a Retry-After header telling the client specifically how long to wait — that value should generally be honored instead of the client's own backoff guess
Classifying an error wrong in the "retry a permanent error" direction wastes attempts and adds load for no benefit; classifying it wrong in the "give up on a transient error" direction surfaces a failure to the user that a simple retry would have fixed
Some errors are ambiguous by nature — a request timeout does not reveal whether the request was never received, was received but not yet processed, or was fully processed and only the response was lost, which is why idempotency matters even for retries of transient-looking failures

Common HTTP status codes, classified for retry purposes

Common HTTP status codes, classified for retry purposes
Status code	Meaning	Retry?
429 Too Many Requests	Rate limited	Yes — ideally honoring a Retry-After header
503 Service Unavailable	Downstream temporarily overloaded or down	Yes — with backoff
504 Gateway Timeout	Upstream did not respond in time	Yes — with backoff, cautiously (see non-idempotent risk)
500 Internal Server Error	Unhandled server-side fault	Sometimes — depends on whether the fault is transient; often treated as retryable with caution
400 Bad Request	Malformed request	No — the same request will fail identically
401 Unauthorized / 403 Forbidden	Auth/permission failure	No — retrying without fixing credentials cannot succeed
404 Not Found	Resource does not exist	No — the resource will not appear from retrying

Transient vs. permanent errors

Transient — retry

429 Too Many Requests, 503, 504
A temporary condition that can resolve itself
Honor Retry-After if the response provides it

Permanent — fail fast

400 Bad Request, 401/403, 404
The identical request fails identically every time
Retrying only adds latency and load, zero upside

Command

text

on error:
    if is_transient(error):
        retry with backoff + jitter, up to max attempts
    else:
        fail fast — surface the error, do not retry

Transient errors (503, 429, timeouts) have a real chance of succeeding on retry; permanent errors (400, 401/403, 404) will fail identically every time. Classify before retrying — retrying a permanent error only adds latency and load for zero chance of success.

4.Set maximum attempts and dead-letter failed messages

A retry policy needs a stopping point. Without a maximum attempt count, a message or request that will never succeed (because the underlying cause is not actually transient, despite looking like it) gets retried forever, consuming resources and hiding the fact that it is permanently stuck. A dead-letter queue (DLQ) is where a message goes after it exhausts its retry attempts without succeeding — instead of being silently dropped or retried forever, it is set aside for inspection, alerting, or manual reprocessing, so the failure is visible and recoverable rather than invisible and lost.

Key features

A maximum attempt count (e.g. 3-5 retries) bounds the worst case — a message that can never succeed eventually stops consuming resources instead of retrying forever
A dead-letter queue (DLQ) is a separate queue or store where a message lands after exhausting its retries — it is neither redelivered nor silently discarded, it becomes visible for inspection
Without a DLQ, a message that exhausts retries either gets dropped silently (losing the failure entirely) or blocks the queue behind it (a poison message that repeatedly fails can stall processing of every message after it)
A DLQ should trigger an alert or be actively monitored — a DLQ nobody looks at is equivalent to silently dropping the message, just with extra storage overhead
Reprocessing a DLQ message after fixing the root cause (a bug, a downstream outage) should generally happen deliberately (a manual or scripted redrive), not automatically — automatic redrive can re-trigger the exact same failure loop

What happens to a message that keeps failing, by policy

What happens to a message that keeps failing, by policy
Policy	Outcome for a message that never succeeds
No max attempts	Retried forever — consumes resources indefinitely, and the queue may stall behind it (a poison message)
Max attempts, no DLQ	Retries stop, but the message is silently dropped — the failure is invisible and unrecoverable
Max attempts + DLQ	Retries stop, the message moves to a separate queue for inspection, alerting and deliberate reprocessing

Bounded retries, then a visible dead-letter queue

checkunder maxexhausted

Message fails

attempt += 1

attempt < max?

backoff + jitter

Retry attempt

requeued

Dead-letter queue

inspection + alert

Command

text

on message failure:
    attempt += 1
    if attempt >= max_attempts:
        send to dead_letter_queue
        alert()
    else:
        retry with backoff + jitter

Cap retry attempts so a message that will never succeed cannot loop forever; route what exhausts its attempts (or fails permanently) to a dead-letter queue instead of dropping it silently — and treat the DLQ as something to monitor and deliberately reprocess, not an automatic redrive target.

5.Do not blindly retry non-idempotent operations

A retry assumes that repeating the exact same request is safe — but a request can fail after the server has already fully processed it, if only the response was lost on the way back (a timeout is genuinely ambiguous: it does not tell the client whether the request was never received, was received but not yet handled, or was fully handled and only the acknowledgment vanished). If the operation is not idempotent, blindly retrying it in that third case repeats the side effect a second time — charging a card twice, sending a duplicate email, incrementing a counter twice for one real event. Retry logic and idempotent operation design are two separate concerns that only become safe together: the fix for idempotency itself lives in the system-design.delivery-semantics-and-idempotency section — this concept is specifically about respecting that boundary from the retry side.

Key features

A timeout is ambiguous by nature: the client cannot distinguish "request never arrived," "request arrived but is still processing," and "request fully succeeded, only the response was lost" — a blind retry treats all three the same way
Retrying is only unconditionally safe for genuinely idempotent operations (a GET, a PUT that sets an absolute value, an operation guarded by a real idempotency key) — retrying a naturally non-idempotent operation (POST that charges a card, increments a counter, sends a notification) without that guard can duplicate the side effect
HTTP methods carry a documented idempotency expectation: GET, PUT, DELETE are specified as idempotent; POST is not — this is exactly why many client libraries default to NOT auto-retrying POST requests without explicit configuration
The fix is not "never retry a POST" — it is "make the operation idempotent first" (an idempotency key, a unique constraint, a deduplication check), then retrying becomes safe; see system-design.delivery-semantics-and-idempotency for how that is actually implemented
Blind retries are especially dangerous for financial and inventory operations specifically because the cost of a duplicate (double charge, double shipment) is high and often not immediately visible to either the client or the user

HTTP methods and their standard idempotency expectation for retry safety

HTTP methods and their standard idempotency expectation for retry safety
Method	Idempotent by spec?	Safe to blindly retry?
GET	Yes	Yes — reading data again has no side effect
PUT	Yes (sets an absolute state)	Yes — applying the same absolute state twice is a no-op the second time
DELETE	Yes	Generally yes — deleting an already-deleted resource is typically a no-op or a harmless 404
POST	No	Not by default — only safe if the operation is explicitly guarded by an idempotency key or equivalent
PATCH (relative change)	No	Not by default — a relative change like "increment by 1" duplicates on retry

Safe to blindly retry, by HTTP method

GET

idempotent by spec

PUT

sets an absolute state

DELETE

usually a no-op repeat

POST

only if idempotency-key guarded

PATCH (relative)

duplicates on retry

Command

text

on request timeout:
    if operation.is_idempotent:
        retry safely
    else:
        do NOT blindly retry —
        first add an idempotency guard, or
        surface the ambiguity instead of guessing

A timeout cannot tell you whether the request actually succeeded — retrying it blindly repeats the side effect if the operation isn't idempotent. GET/PUT/DELETE are safe to retry by default; POST and relative updates are not, unless explicitly guarded by an idempotency key or equivalent (see system-design.delivery-semantics-and-idempotency for the implementation mechanisms).

Spotted a mistake on this page? Tell us.

Read Sheet