beginnerintermediateadvanced

System Design · Circuit Breakers and Bulkheads

Circuit Breakers and Bulkheads

Failing fast instead of failing slow, and containing damage so it cannot spread

Fail faststop calling a known-failing dependency
Three statesclosed → open → half-open → closed
Bulkheadsdedicated resource slice per dependency
Layered togetherfour techniques, four different questions

1.Why circuit breakers exist: stop hammering a failing dependency

A remote call can fail outright or hang until a timeout is reached — either way, every caller that keeps trying it wastes resources (threads, connections, time) on a call that is unlikely to succeed. A circuit breaker wraps a call to a dependency and, after enough recent failures, stops attempting the call at all for a while — failing fast locally instead of repeatedly waiting on a call that keeps failing. This protects the caller from wasting its own resources, and protects the struggling dependency from a continuous stream of load that makes recovery harder, not easier.

Key features

Without a circuit breaker, every caller of a failing dependency keeps trying, each attempt tying up a thread, connection or other resource until its own timeout fires — under load, this can exhaust the caller's own capacity purely from waiting on a dependency that was never going to succeed
A circuit breaker tracks recent call outcomes (success/failure) for a specific dependency and, once failures cross a threshold, "trips" and stops attempting real calls for a cooldown period
Failing fast (an immediate local error when the breaker is tripped) is dramatically cheaper than failing slow (waiting out a full timeout on every attempt) — the caller finds out sooner and can react (fallback, queue, surface an error) instead of blocking
A circuit breaker protects in two directions at once: it protects the caller from wasting resources on a call unlikely to succeed, and it protects the struggling dependency from continued load that can make its own recovery slower or impossible
Circuit breakers are typically scoped per dependency (or per specific downstream call), not globally — one failing dependency tripping its own breaker should not affect calls to a healthy, unrelated dependency

With vs without a circuit breaker, same failing dependency

With vs without a circuit breaker, same failing dependency
Property	Without a breaker	With a breaker (tripped)
Time to find out a call failed	Full timeout, every single attempt	Immediate — no real call attempted
Load placed on the failing dependency	Continuous, from every caller, every attempt	None, while the breaker is open
Caller resource usage	A thread/connection held per in-flight failing call	Freed immediately to do other work
Recovery conditions for the dependency	Actively worse — recovering under continued full load	Better — no incoming load to fight through while recovering

Calling a failing dependency, with vs. without a breaker

Without a breaker

Full timeout paid on every single attempt
Continuous load on the failing dependency
A thread/connection held per in-flight call

With a breaker (tripped)

Immediate failure — no real call attempted
Zero load while the breaker is open
Caller resources freed immediately

Command

text

if breaker.is_tripped(dependency):
    return fail_fast()          # no real call attempted
else:
    result = call(dependency)
    breaker.record(result)      # feeds the trip decision
    return result

A circuit breaker stops a caller from repeatedly attempting a call to a dependency that recent evidence shows is failing — failing fast instead of failing slow, which protects both the caller's own resources and gives the struggling dependency room to actually recover.

2.The closed, open and half-open states

A circuit breaker is a three-state machine. Closed is normal operation — calls go through, and successes reset any accumulated failure count. Once failures cross a configured threshold, the breaker trips to Open — every call fails immediately with no real attempt made, for a configured cooldown period. After that cooldown elapses, the breaker moves to Half-Open — a small number of trial calls are allowed through to test whether the dependency has actually recovered. A successful trial call closes the breaker again (back to normal); a failed trial call reopens it and restarts the cooldown.

Key features

Closed: normal state — calls are attempted, successes are recorded, and a failure counter (or failure rate over a rolling window) accumulates on failures
Closed → Open: triggered once the failure count or failure rate crosses a configured threshold within a configured window (e.g. "5 failures in 10 seconds" or "50% failure rate over the last 20 calls")
Open: calls fail immediately with no real attempt made against the dependency at all — this is the state that actually stops load from reaching a struggling dependency
Open → Half-Open: triggered automatically once a configured cooldown/reset timeout elapses — not by any external signal, the breaker just tries again on a timer
Half-Open: a limited number of trial calls are allowed through; a successful trial (or a configured number of consecutive successes) transitions back to Closed, while a single failed trial (in the simplest implementations) transitions back to Open and restarts the cooldown

The three states and what happens in each

The three states and what happens in each
State	Real calls attempted?	Trigger to leave this state
Closed	Yes, all of them	Failure threshold crossed → Open
Open	No — fail fast immediately	Cooldown timeout elapses → Half-Open
Half-Open	Yes, a limited trial number	Trial succeeds → Closed; trial fails → Open

The circuit breaker state machine

thresholdcrossedcooldownelapsestrialsucceedstrial fails

Closed

start

Open

Half-Open

Closed again

end

Cooldown restarts

end

Command

text

Closed --[failures >= threshold]--> Open
Open --[cooldown elapses]--> Half-Open
Half-Open --[trial succeeds]--> Closed
Half-Open --[trial fails]--> Open

Closed: normal, calls attempted. Open: fail fast, no real calls, triggered by a failure threshold. Half-Open: a limited trial after the cooldown elapses — success returns to Closed, failure returns to Open and restarts the cooldown.

3.Bulkheads isolate resources so one workload cannot consume everything

A bulkhead partitions a shared resource (a thread pool, a connection pool, a rate-limit budget) into isolated slices, one per workload or dependency, so that one workload consuming all of its own slice cannot also consume the slices allocated to other workloads. The name comes directly from a ship's bulkheads — the watertight compartment walls that keep a hull breach in one compartment from flooding the entire ship. Applied to software, a slow or failing call to Dependency A that exhausts its own dedicated thread pool leaves Dependency B's separate pool completely untouched, so calls to B keep working normally.

Key features

Bulkhead pattern: partition a shared resource (thread pool, connection pool, semaphore) into separate, dedicated allocations per workload or dependency, instead of one pool shared by everything
The isolation is what matters — a workload that exhausts its own bulkhead's capacity is degraded or fails on its own, but every other bulkhead (and the workloads depending on them) is unaffected
Common implementation: a fixed-size thread pool or semaphore per downstream dependency, so a slow dependency can only ever block up to that dependency's own pool size worth of concurrent callers, never the whole service's total capacity
Bulkheads trade some efficiency for isolation — a bulkhead sized for dependency A sits partially idle if A is quiet, even while dependency B's bulkhead is saturated, because capacity cannot be borrowed across bulkheads by design
Bulkheads and circuit breakers solve related but distinct problems: a circuit breaker stops calling a failing dependency; a bulkhead limits how much of a shared resource any single dependency (failing or not) can ever consume — they are commonly used together, not as alternatives

One shared pool vs bulkheaded pools, same total capacity

One shared pool vs bulkheaded pools, same total capacity
Property	One shared pool (no bulkhead)	Bulkheaded (per-dependency pools)
Effect of dependency A hanging	Can consume the entire shared pool, starving calls to B, C, D	Bounded to A's own dedicated slice; B, C, D unaffected
Resource efficiency	Higher — no idle capacity sitting unused in another workload's slice	Lower — a quiet dependency's slice sits partially idle
Failure isolation	None — one dependency's failure can take down calls to every other dependency	Strong — failure is contained to the one dependency's own bulkhead

Bulkheaded thread pools, one per dependency

recommendations_pool (30)

blocked

search_pool (70)

healthy

Command

text

# instead of one shared pool:
shared_pool = ThreadPool(size=100)   # any dependency
                                       # can consume all 100

# bulkheaded: dedicated pool per dependency
payments_pool = ThreadPool(size=20)
search_pool   = ThreadPool(size=30)
email_pool    = ThreadPool(size=10)
# a hung payments call can block at most 20 threads,
# never touching search's or email's own capacity

A bulkhead partitions a shared resource into dedicated, isolated slices per workload or dependency — named for a ship's watertight compartments — so one workload exhausting its own slice cannot also consume another workload's allocation. Size each slice from that dependency's real measured demand, and combine bulkheads with circuit breakers rather than choosing one over the other.

4.Combining circuit breakers, bulkheads, timeouts and bounded retries

None of the four resilience techniques covered so far — timeouts, bounded retries, circuit breakers, bulkheads — substitutes for the others, because each answers a different question about the same failing call. A timeout answers "how long do I wait for one attempt." A bounded retry policy answers "how many times do I try before giving up." A circuit breaker answers "should I even attempt this call at all, given recent history." A bulkhead answers "how much of my own resources can this dependency ever consume, even while all of the above are still deciding." A resilient call to an external dependency typically uses all four together, layered around the same call.

Key features

Order of application on a single call: a timeout bounds each individual attempt; a bounded retry policy governs how many attempts are made using that timeout; a circuit breaker sits above the retry policy and can skip the whole attempt-and-retry cycle entirely once tripped; a bulkhead bounds how much of the caller's own resource pool the whole cycle can ever consume
A circuit breaker without bounded retries still lets each individual attempt (before the breaker trips) retry an unbounded number of times, defeating the breaker's own purpose of limiting wasted effort
Bounded retries without a circuit breaker still retry every single new incoming request independently against a dependency already known to be failing, even though the breaker's whole purpose is recognizing that pattern across requests, not just within one
A bulkhead without a circuit breaker still isolates one dependency's resource consumption from others, but that dependency's own dedicated pool can still be fully consumed by unbounded retries against it, if nothing stops the retries themselves
The combination has a natural order: bulkhead scopes the resource, circuit breaker decides whether to even attempt, bounded retry governs how many attempts, timeout bounds each individual attempt — each layer catches what the others structurally cannot

What each technique answers, and what it does not cover alone

What each technique answers, and what it does not cover alone
Technique	Question it answers	What it does not cover alone
Timeout	How long to wait for one attempt	How many attempts, or whether to attempt at all
Bounded retry	How many attempts before giving up	Whether a whole dependency is known to be failing across requests
Circuit breaker	Whether to even attempt this call, given recent history	How much shared resource an in-flight attempt can consume
Bulkhead	How much of a shared resource this dependency can ever consume	When to actually stop attempting a specific failing call

Four layered defenses, four different questions

Bulkhead

how much resource this can consume

Circuit breaker

whether to attempt at all

Bounded retry

how many attempts

Timeout

how long each attempt waits

Command

text

bulkhead(dependency_pool):
  if breaker.is_tripped(dependency):
    return fail_fast()
  for attempt in range(max_retries):
    try:
      return call(dependency, timeout=deadline)
    except TransientError:
      wait(backoff(attempt))
  breaker.record(failure)
  return give_up()

Timeouts, bounded retries, circuit breakers and bulkheads answer four different questions about the same failing call and are meant to be layered together, not chosen between — a bulkhead bounds resource consumption, a circuit breaker decides whether to attempt at all, bounded retries govern how many attempts, and a timeout bounds each individual attempt.

Spotted a mistake on this page? Tell us.

Read Sheet