beginnerintermediateadvanced

System Design · Section 75

Cascading Failures

Level: intermediate
Read: 12 min
Concepts: 2

A cascading failure is defined by a mismatch: the eventual outage is far larger than the original trigger, because every cause behind it works by amplification rather than by the trigger itself being large. A retry storm turns the fix (retrying a failed call) into the cause of continued failure, by adding load to a dependency that is already struggling. Shared resource exhaustion lets one call path's slowdown starve every other path that happens to draw from the same connection or thread pool, regardless of whether they are related. An overloaded database — often the one resource many independent services genuinely share — turns an isolated slow-query problem into everyone's problem at once. A queue backlog is self-reinforcing: as it grows, processing gets slower, which grows the backlog faster, with no natural point where it stabilizes on its own. And synchronized clients turn load that should be smoothly distributed into a sharp, simultaneous spike, purely because many independent clients coincidentally act on the same schedule. Every mitigation targets one of these amplification paths directly rather than the original trigger: bounded retries and jitter cap and desynchronize the retry reaction, circuit breakers stop calls to an already-failing dependency, admission control and load shedding limit how much new work enters an overloaded system, backpressure signals a producer to slow down before a backlog forms, and bulkheads remove the shared pool that let one path's slowdown spill into an unrelated one — each mechanism already covered in full elsewhere in this topic, with this section's own job being to map which mitigation closes which cause, since a resilient design needs several of them layered together, not just one.

This section

How to avoid cascading failures in a distributed system 💣💥🔥Gaurav Sen

What is true here

A cascading failure's signature is amplification: the eventual outage is far larger than the original, often small, trigger.
Five common causes: retry storms, shared resource exhaustion, overloaded databases, queue backlogs, synchronized clients — each amplifies a contained problem into a widespread one.
Seven mitigations — bounded retries, jitter, circuit breakers, admission control, backpressure, load shedding, bulkheads — each cap one specific amplification path.
A circuit breaker and bounded retries are complementary, not redundant — one controls whether a call is attempted, the other controls what happens inside a call that is.
No single mitigation is sufficient; a real cascade usually compounds more than one cause, so a resilient design layers several mitigations together.

What you will be able to do

Name the five common causes of cascading failure and explain the amplification mechanism behind each one
Trace an incident timeline and identify which cause is compounding at each stage
Map each of the seven standard mitigations to the specific cause(s) it targets
Explain why fixing only the cause of the last incident does not prevent the next, differently-caused cascade

How cascades start

Five causes, one shared mechanism: a locally reasonable, unbounded reaction amplifying a small problem into a large one.

Common causes of cascading failure

coreintermediate

A cascading failure is one where a small, initially contained problem in one component spreads to bring down other, otherwise-healthy components — the defining feature is that the system ends up in a much worse state than the size of the original trigger would suggest. Five causes account for most real-world cascades. A retry storm happens when a dependency slows down or briefly fails, every caller retries (often simultaneously), and the retries themselves add enough extra load to keep the dependency down or push previously-healthy dependencies down too — the "fix" (retrying) becomes the cause of continued failure. Shared resource exhaustion happens when multiple unrelated call paths draw from the same pool (connections, threads, memory) and one path's slowdown consumes the entire shared pool, starving every other path that happens to share it. An overloaded database is a common single point where many independent services converge, so a spike or slow-query problem in the database becomes everyone's problem simultaneously, rather than being contained to whichever service caused it. Queue backlogs occur when a consumer falls behind a producer's rate; the backlog itself starts consuming memory or disk, and as the backlog grows, message processing and lookups against it get slower, which slows the consumer further and grows the backlog even faster — a self-reinforcing loop. Synchronized clients happen when many independent clients end up doing the same thing at the same moment (all retrying on the same fixed schedule, all refreshing a cache on the same TTL, all waking from the same cron), turning what should be smoothly-distributed load into a sharp, simultaneous spike.

Think of it as

A cascading failure is a traffic jam that started with one stalled car. The stalled car alone (the original, small failure) blocks one lane; but cars behind it brake hard (a retry storm — everyone reacting to the same event at once), the merging traffic backs up into an earlier intersection that had nothing to do with the original stall (shared resource exhaustion — an unrelated route now blocked too), and within twenty minutes an entire section of the highway network is gridlocked from a single stalled car that could have been towed in five minutes if nothing else had reacted to it so aggressively. The size of the eventual jam has almost no relationship to the size of the original problem — that mismatch is the actual signature of a cascading failure, as opposed to an outage that stays proportional to its cause.

text

# A cascade traced through all five causes in
# roughly the order they compound:
1. Database has a slow-query spike (root cause,
   small on its own)
2. Every service querying it (overloaded database)
   sees rising latency
3. Callers time out and retry (retry storm) --
   the retries add MORE load to the already-slow DB
4. Retrying callers share a connection pool with
   unrelated calls (shared resource exhaustion) --
   unrelated calls start failing too
5. A queue in front of an async consumer of the
   same DB starts backing up (queue backlog) as
   consumers slow down
6. Client-side cache TTLs, all set to the same
   round value, expire simultaneously
   (synchronized clients) -- a wave of cache
   misses hits the DB right as it is recovering

What we're doing: Trace how a brief database blip becomes a full outage through retry amplification and shared resources.

incident-timeline.txttext

00:00  A missing index causes one query pattern
       to take 4s instead of 40ms (root cause)
00:01  Callers of that query start timing out and
       retrying immediately with no backoff
00:02  Retry volume roughly doubles total query
       load against the database
00:03  The connection pool shared by 6 unrelated
       services against this DB is fully consumed
       by retrying + slow requests
00:04  All 6 services start failing, including 4
       that never queried the slow pattern at all
00:06  A queue in front of an async worker (which
       also depends on the DB) starts backing up
00:15  Full outage across services with no direct
       relationship to the original missing index

3: Immediate, unbounded retry is what converts a 4-second slow query into a doubled load spike within one minute — the retry storm is not a separate incident, it is the mechanism that turns a contained slowdown into a load problem.
7: This is the moment the cascade crosses from "the slow-query services are degraded" to "unrelated services are down too" — a shared connection pool is what makes that crossing possible; without it, the four unrelated services in the next line would have stayed healthy.

Why this works: Every individual step in this timeline is a small, locally reasonable behavior — retrying a timed-out call, sharing a connection pool to save resources, letting a queue absorb temporary slowness — and the outage is not caused by any single bad decision, but by the fact that none of those locally reasonable behaviors had a limit, so each one amplified rather than absorbed the original problem.

Retrying immediately and unconditionally on any failure

Wrong

python

def query_with_retry(sql):
    while True:
        try:
            return db.execute(sql)
        except (Timeout, ConnectionError):
            continue  # retry immediately, forever

Better

python

def query_with_retry(sql, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return db.execute(sql)
        except (Timeout, ConnectionError):
            if attempt == max_attempts - 1:
                raise
            sleep(backoff_with_jitter(attempt))
    raise ExhaustedRetries()

What you see: A database that was recovering from a brief slow-query spike never actually recovers, because every caller's unconditional retry loop keeps resending failed queries as fast as they fail, holding the database at saturation indefinitely instead of letting it drain the backlog and return to normal.

Why: An unbounded, immediate retry loop has no mechanism to reduce load even when load is exactly the problem — it treats every failure as "try again right now," which is the correct instinct for a single caller in isolation and the exact wrong aggregate behavior when thousands of callers all reach the same conclusion about the same struggling dependency simultaneously.

How one slow query becomes a system-wide outage

DB slow-query spike

small, isolated

Overloaded database

every caller feels it

Retry storm

retries add more load

Shared resource exhaustion

unrelated calls starved too

Queue backlog

self-reinforcing slowdown

System-wide outage

far larger than the original spike

Five causes and the amplification mechanism each one relies on

Five causes and the amplification mechanism each one relies on
Cause	Amplification mechanism	What normally contains it
Retry storm	Retries add load to an already-struggling dependency	Bounded retries with backoff + jitter, circuit breakers
Shared resource exhaustion	One slow path consumes a pool shared by unrelated paths	Bulkheads (dedicated pools per dependency)
Overloaded database	Many independent services converge on one resource	Connection limits, read replicas, caching, query timeouts
Queue backlog	Growing backlog slows processing, which grows the backlog further	Backpressure, load shedding, consumer autoscaling
Synchronized clients	Independent clients act at the same instant by coincidence	Jitter on schedules/TTLs, staggered rollouts

Remember: Five causes account for most cascading failures — retry storms, shared resource exhaustion, overloaded databases, queue backlogs, and synchronized clients — and every one of them works by amplification: a locally reasonable reaction (retry, share a pool, let a queue absorb slack) turns a small, contained problem into a much larger one because nothing limited how far the reaction could go.

Stopping the amplification

Seven mitigations, each capping a specific amplification path — mapped to cause, not re-derived from scratch.

Mitigating cascading failures

coreintermediate

Every cause of cascading failure covered in the prior concept works by amplification — a locally reasonable reaction with no limit turning a small problem into a large one — so every mitigation works the same way in reverse: put an explicit limit on the reaction so it cannot amplify past a bounded point. Seven mechanisms cover the standard toolkit, each already given full treatment elsewhere in this topic, so this concept's job is naming which cause each one targets rather than re-explaining the mechanism itself. Bounded retries (a maximum attempt count) and jitter (randomizing retry timing so callers do not all retry in lockstep) directly target retry storms and synchronized clients. Circuit breakers target retry storms and overloaded databases by stopping calls to a dependency that is already failing, which both protects the caller and reduces load on the struggling dependency. Admission control (rejecting new work up front once a system is at capacity, rather than accepting it and failing later) and load shedding (a specific form of admission control: intentionally dropping lower-priority work under overload) target overloaded databases and queue backlogs by capping how much work enters the system at all. Backpressure (a consumer signaling upstream to slow down rather than silently falling behind) directly targets queue backlogs by preventing the self-reinforcing growth loop before it starts. Bulkheads target shared resource exhaustion directly, by removing the shared pool that let one path's slowdown starve an unrelated one. No single mechanism covers every cause — a resilient design layers several of these together, because a real cascade rarely has just one cause acting alone.

Think of it as

If a cascading failure is a traffic jam that grew from one stalled car because every reaction (hard braking, rerouting into an already-busy street, everyone checking the same traffic app at once) added more congestion, then the mitigations are traffic engineering: a merge limiter that only lets cars onto the highway at a sustainable rate (admission control), dedicated lanes so a jam in the general lanes cannot spill into the bus lane (bulkheads), a sign that tells drivers behind the stall to take an alternate route rather than idling and blocking others (circuit breaker), staggered light timing so cars from different streets do not all arrive at the same intersection simultaneously (jitter), and a rule that emergency vehicles get priority when the road is over capacity while ordinary traffic waits (load shedding by priority). None of these prevent the original stalled car; they prevent it from becoming a city-wide gridlock.

text

# The same incident from the prior concept,
# with mitigations layered in at each stage:
1. DB slow-query spike               -> (root cause;
                                          fix the index)
2. Callers see rising latency        -> circuit breaker
                                          opens, stops
                                          sending traffic
3. Callers would retry               -> bounded retries
                                          + jitter cap
                                          the added load
4. Shared connection pool            -> bulkhead: each
                                          service has its
                                          own pool
5. Queue in front of async worker    -> backpressure
   starts to grow                       signals producer
                                          to slow down
6. DB still over capacity            -> admission control
                                          rejects new
                                          non-critical
                                          queries; load
                                          shedding drops
                                          lowest-priority
                                          work first

What we're doing: Apply the seven mitigations to the same database-slowdown incident traced in the prior concept, and compare the outcome.

mitigated-incident-timeline.txttext

00:00  A missing index causes one query pattern
       to take 4s instead of 40ms (same root cause)
00:01  Callers' circuit breakers trip after a short
       error-rate threshold -- most calls fail fast
       instead of piling onto the slow DB
00:01  The retries that do happen are bounded (3
       attempts) and jittered, so they don't spike
       in lockstep
00:02  Each service's calls to this DB use their
       own connection pool (bulkhead) -- 4 unrelated
       services are entirely unaffected
00:03  The async worker's queue starts backpressure
       -- its producer slows down before a backlog
       forms
00:04  Admission control rejects new non-critical
       report queries against the DB; load shedding
       drops the lowest-priority background jobs first
00:10  DB recovers; circuit breakers close; traffic
       resumes normally -- outage stayed contained
       to the originally-affected query pattern

3: This is the single biggest difference from the unmitigated timeline: instead of every caller retrying and adding load, the circuit breaker converts "the DB is struggling" into "most calls fail immediately," which is a far smaller total load on the database than a wave of retries would have been.
9: Because each service has its own connection pool, the 4 services unrelated to the slow query pattern never even notice the incident — this is the direct payoff of paying the bulkhead's setup cost before the incident, not during it.

Why this works: Nothing about the root cause changed between the two timelines — the same missing index causes the same 4-second query — but the mitigated version resolves in 10 minutes with a contained blast radius, while the unmitigated version spread into a full outage, which is the entire point: these mechanisms do not prevent the trigger, they prevent the trigger from being amplified into something far worse.

Adding circuit breakers everywhere but leaving retries unbounded

Wrong

python

@circuit_breaker(threshold=0.5)
def call_dependency():
    while True:               # circuit breaker wraps
        try:                   # the call, but retries
            return dependency.call()  # inside it are
        except ServiceError:   # still unbounded
            continue

Better

python

@circuit_breaker(threshold=0.5)
def call_dependency():
    for attempt in range(3):   # bounded, and jittered
        try:
            return dependency.call()
        except ServiceError:
            if attempt == 2:
                raise
            sleep(backoff_with_jitter(attempt))

What you see: Even after a circuit breaker is added, the dependency stays overloaded during an incident, because every call that gets through before the breaker opens still retries in an unbounded loop, and the breaker's own periodic "try again" probes compound with those unbounded retries.

Why: A circuit breaker controls whether a call is attempted at all; it does nothing about what happens inside a call that is attempted — the two mechanisms solve different, complementary parts of the problem, and adding one without the other leaves the exact amplification path the missing one was supposed to close.

Seven mitigations, grouped by what they act on

Slow the reaction down

Bounded retries

cap attempt count

Jitter

desynchronize retry timing

Stop calling a struggling dependency

Circuit breaker

fail fast instead of piling on

Limit what enters the system

Admission control

reject new work at the door

Load shedding

drop lowest-priority work first

Backpressure

signal upstream to slow down

Isolate the blast radius

Bulkheads

separate pools per dependency

Seven mitigations mapped to the cause each one targets

Seven mitigations mapped to the cause each one targets
Mitigation	Cause(s) it targets	Covered in full at
Bounded retries	Retry storms	Retry Strategy — Max Attempts and Dead-Lettering
Jitter	Retry storms, synchronized clients	Retry Strategy — Backoff and Jitter
Circuit breakers	Retry storms, overloaded databases	Circuit Breakers and Bulkheads
Admission control	Overloaded databases, queue backlogs	Load Shedding (§77)
Backpressure	Queue backlogs	Backpressure
Load shedding	Overloaded databases, queue backlogs	Load Shedding (§77)
Bulkheads	Shared resource exhaustion	Circuit Breakers and Bulkheads — Bulkhead Isolation

Remember: Seven mechanisms — bounded retries, jitter, circuit breakers, admission control, backpressure, load shedding, bulkheads — each target a specific cause of cascading failure, and each is already covered in full elsewhere in this topic. No single one is sufficient alone; a resilient design layers several together, matched to which causes are actually present, rather than adding one point fix per past incident.