beginnerintermediateadvanced

System Design · Section 33

Circuit Breakers and Bulkheads

Level: intermediate
Read: 22 min
Concepts: 4

A circuit breaker wraps a call to a dependency and, once recent failures cross a threshold, stops attempting the call at all — failing fast locally instead of every caller independently waiting out a full timeout on a call already known to be failing. It moves through three states: Closed (normal, calls attempted), Open (fail immediately, no real attempt made, triggered by a failure threshold), and Half-Open (a limited trial after a cooldown, returning to Closed on success or back to Open on failure). A bulkhead — named after a ship's watertight compartments — partitions a shared resource like a thread pool into dedicated slices per dependency, so one dependency exhausting its own slice cannot also consume capacity that belongs to another. Neither technique substitutes for timeouts or bounded retries; all four answer different questions about the same failing call and are meant to be layered together.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

A circuit breaker stops repeatedly attempting a call to a dependency that recent evidence shows is failing, protecting both the caller's resources and the dependency's ability to recover.
The state machine is Closed (normal) → Open (fail fast, triggered by a failure threshold) → Half-Open (one limited trial after a cooldown) → back to Closed or Open.
A bulkhead partitions a shared resource into dedicated per-dependency slices so one dependency's failure cannot starve another's capacity.
Timeouts, bounded retries, circuit breakers and bulkheads each answer a different question and are combined together, not chosen between.

What you will be able to do

Explain why a circuit breaker is a different defense than a timeout, not a redundant one
Trace a circuit breaker through a full trip-and-recover cycle across its three states
Design a bulkhead sized from a dependency's real measured demand rather than an arbitrary equal split
Layer timeouts, bounded retries, circuit breakers and bulkheads correctly around the same call

The circuit breaker

Why it exists, and the three-state machine every implementation follows.

Why circuit breakers exist: stop hammering a failing dependency

coreintermediate

A remote call can fail outright or hang until a timeout is reached — either way, every caller that keeps trying it wastes resources (threads, connections, time) on a call that is unlikely to succeed. A circuit breaker wraps a call to a dependency and, after enough recent failures, stops attempting the call at all for a while — failing fast locally instead of repeatedly waiting on a call that keeps failing. This protects the caller from wasting its own resources, and protects the struggling dependency from a continuous stream of load that makes recovery harder, not easier.

Think of it as

It is named after the electrical circuit breaker in a building for a reason. When a circuit is overloaded, the breaker trips and cuts power immediately, rather than letting every appliance keep drawing current into a fault and risking a fire. A software circuit breaker does the same thing to a failing dependency: once it detects a fault, it "trips" and stops sending more calls through, protecting the caller from wasting effort and giving the dependency room to actually recover, rather than being hit by an unrelenting stream of retries the moment it shows the first sign of trouble.

text

if breaker.is_tripped(dependency):
    return fail_fast()          # no real call attempted
else:
    result = call(dependency)
    breaker.record(result)      # feeds the trip decision
    return result

What we're doing: Show the resource-exhaustion cascade a missing circuit breaker allows, and how a breaker prevents it.

no-breaker-cascade.txttext

Service A calls Service B for every incoming request.
Service B has degraded and now takes 30s to respond
to every call (its own downstream dependency died).

Service A has a thread pool of 100 worker threads,
no circuit breaker on its call to B.

t=0    100 requests arrive at A, each calling B.
       Each of A's 100 threads is now blocked
       waiting up to 30s for B to respond.
t=1    101st request arrives. A has ZERO free
       threads left — this request queues, or is
       rejected outright, even though it has
       nothing to do with B at all.
t=1-30 A is now effectively down for ALL traffic,
       not just the traffic that needed B, because
       every thread is tied up waiting on B's slow
       responses.

8: This is the moment the cascade begins — A's entire worker pool is consumed by calls to one slow dependency.
13: The failure has now spread past its origin: A is unavailable for ALL requests, including ones that never touch B, purely because of thread exhaustion.

Why this works: This is the exact failure a circuit breaker on the call to B prevents — once B's failure rate crosses the trip threshold, A stops calling B at all, freeing every thread that would otherwise be stuck waiting, and A stays available for everything that does not depend on B.

Assuming a timeout alone prevents the cascading-failure scenario a circuit breaker exists for

Wrong

text

"We have a 5-second timeout on calls to B, so
we're protected — worst case, a thread is only
tied up for 5 seconds."

Better

text

"A 5-second timeout bounds ONE call, but under
sustained load every new request still spends up
to 5 seconds blocked on a call to a dependency
we already know is failing. A circuit breaker
fails those calls in microseconds once B's
failure rate crosses the threshold, instead of
letting every single request pay the full
timeout independently."

What you see: A service with a "reasonable" per-call timeout still experiences a full outage under sustained load from one failing dependency — the timeout bounds each individual call, but does nothing to stop every new incoming request from independently paying that same bounded cost, which at high enough request volume still exhausts the caller's capacity.

Why: A timeout answers "how long should I wait for one call," while a circuit breaker answers a different question entirely — "should I even attempt this call at all, given how the last N attempts to this same dependency went" — and only the second question actually stops a caller from repeatedly paying the timeout cost for calls it already has strong evidence will fail.

Calling a failing dependency, with vs. without a breaker

Without a breaker

+Full timeout paid on every single attempt
+Continuous load on the failing dependency
+A thread/connection held per in-flight call

With a breaker (tripped)

−Immediate failure — no real call attempted
−Zero load while the breaker is open
−Caller resources freed immediately

With vs without a circuit breaker, same failing dependency

With vs without a circuit breaker, same failing dependency
Property	Without a breaker	With a breaker (tripped)
Time to find out a call failed	Full timeout, every single attempt	Immediate — no real call attempted
Load placed on the failing dependency	Continuous, from every caller, every attempt	None, while the breaker is open
Caller resource usage	A thread/connection held per in-flight failing call	Freed immediately to do other work
Recovery conditions for the dependency	Actively worse — recovering under continued full load	Better — no incoming load to fight through while recovering

Remember: A circuit breaker stops a caller from repeatedly attempting a call to a dependency that recent evidence shows is failing — failing fast instead of failing slow, which protects both the caller's own resources and gives the struggling dependency room to actually recover.

The closed, open and half-open states

coreintermediate

A circuit breaker is a three-state machine. Closed is normal operation — calls go through, and successes reset any accumulated failure count. Once failures cross a configured threshold, the breaker trips to Open — every call fails immediately with no real attempt made, for a configured cooldown period. After that cooldown elapses, the breaker moves to Half-Open — a small number of trial calls are allowed through to test whether the dependency has actually recovered. A successful trial call closes the breaker again (back to normal); a failed trial call reopens it and restarts the cooldown.

Think of it as

It is like deciding whether to keep calling a friend who has not been picking up. Closed is calling normally, no hesitation, because they usually answer. After enough unanswered calls in a row, you switch to Open — you stop calling altogether for a while, because continuing to call clearly is not working and just wastes your own time. After some time has passed, you move to Half-Open — you try calling exactly once to check if they are answering again, without immediately going back to calling constantly. If that one call gets through, you resume calling normally (back to Closed); if it does not, you go quiet again for another stretch (back to Open) rather than assuming one missed call means they are answering again.

text

Closed --[failures >= threshold]--> Open
Open --[cooldown elapses]--> Half-Open
Half-Open --[trial succeeds]--> Closed
Half-Open --[trial fails]--> Open

What we're doing: Walk through a full trip-and-recover cycle with concrete threshold and cooldown values.

breaker-lifecycle.txttext

Config: trip after 5 consecutive failures,
        cooldown 30 seconds, 1 trial call on
        half-open.

t=0s   State: Closed. Calls succeeding normally.
t=10s  Dependency starts failing. 5 consecutive
       failures accumulate by t=12s.
t=12s  State: Closed -> Open. Every call from
       t=12s onward fails immediately, no real
       attempt reaches the dependency.
t=42s  30-second cooldown elapses.
       State: Open -> Half-Open.
t=42s  Exactly one trial call is allowed through.

Scenario A: trial succeeds
t=42s  State: Half-Open -> Closed. Normal calls
       resume immediately.

Scenario B: trial fails (dependency still down)
t=42s  State: Half-Open -> Open. Cooldown restarts;
       next trial attempt at t=72s, not t=42s again.

9: This is the state that actually stops load — from this moment, no real call reaches the dependency until the cooldown elapses.
15: The single trial call is the entire test — the breaker does not resume full traffic speculatively, it checks first with minimal exposure.

Why this works: The Half-Open state's single-trial-call design is deliberate — it lets the breaker test recovery without immediately re-exposing the dependency to full load again, which would risk re-triggering the exact failure the breaker just protected against.

Sending full traffic volume immediately when transitioning to half-open, instead of a limited trial

Wrong

text

# on cooldown elapse, resume ALL traffic
# immediately as if fully closed
if cooldown_elapsed:
    state = "closed"  # skips half-open entirely

Better

text

if cooldown_elapsed:
    state = "half-open"
    # allow only 1 (or a small fixed number of)
    # trial call(s) through; keep failing fast
    # for everything else until the trial result
    # is known

What you see: A dependency that was recovering slowly gets hit with a full burst of resumed traffic the instant the cooldown timer expires, which can overwhelm it again before it has actually finished recovering — the breaker trips right back open, and the cycle repeats without ever giving the dependency a real chance to stabilize under light load first.

Why: Skipping half-open and jumping straight back to full traffic defeats the actual purpose of a graduated recovery check — a struggling dependency that is 90% recovered can still be knocked back down by a sudden full-volume resumption, whereas a single trial call (or a small number) tests recovery without risking that exact re-injury.

The circuit breaker state machine

Closed

start

Open

Half-Open

Closed again

end

Cooldown restarts

end

The three states and what happens in each

The three states and what happens in each
State	Real calls attempted?	Trigger to leave this state
Closed	Yes, all of them	Failure threshold crossed → Open
Open	No — fail fast immediately	Cooldown timeout elapses → Half-Open
Half-Open	Yes, a limited trial number	Trial succeeds → Closed; trial fails → Open

Remember: Closed: normal, calls attempted. Open: fail fast, no real calls, triggered by a failure threshold. Half-Open: a limited trial after the cooldown elapses — success returns to Closed, failure returns to Open and restarts the cooldown.

Resource isolation, and combining all four defenses

Bulkheads as a separate, resource-focused isolation technique, and how all four resilience patterns fit together around one call.

Bulkheads isolate resources so one workload cannot consume everything

coreintermediate

A bulkhead partitions a shared resource (a thread pool, a connection pool, a rate-limit budget) into isolated slices, one per workload or dependency, so that one workload consuming all of its own slice cannot also consume the slices allocated to other workloads. The name comes directly from a ship's bulkheads — the watertight compartment walls that keep a hull breach in one compartment from flooding the entire ship. Applied to software, a slow or failing call to Dependency A that exhausts its own dedicated thread pool leaves Dependency B's separate pool completely untouched, so calls to B keep working normally.

Think of it as

A cargo ship's hull is divided into separate watertight compartments by bulkhead walls specifically so that a breach in one compartment — a hole below the waterline — floods only that one section and not the entire ship. Without those walls, water entering anywhere could spread throughout the hull and sink the vessel from a single point of damage. A software bulkhead does the same thing to a shared resource pool: partition it so that one workload "taking on water" (failing, running slow, consuming its allocation) stays contained to its own compartment instead of sinking every other workload sharing what used to be one big, undivided resource.

text

# instead of one shared pool:
shared_pool = ThreadPool(size=100)   # any dependency
                                       # can consume all 100

# bulkheaded: dedicated pool per dependency
payments_pool = ThreadPool(size=20)
search_pool   = ThreadPool(size=30)
email_pool    = ThreadPool(size=10)
# a hung payments call can block at most 20 threads,
# never touching search's or email's own capacity

What we're doing: Show a shared thread pool letting one slow dependency starve an unrelated one, and the bulkhead fix.

bulkhead-vs-shared-pool.txttext

Service has 2 downstream dependencies: "search"
(usually fast) and "recommendations" (a third-party
API that just started hanging on every call).

WITHOUT a bulkhead: 1 shared pool of 100 threads.

t=0   recommendations starts hanging on every call.
      Threads calling it pile up, each stuck until
      its own timeout fires.
t=5   80 of the 100 shared threads are now blocked
      on recommendations calls.
t=6   A burst of search requests arrives. Only 20
      threads are free — search, which has nothing
      to do with recommendations, now queues and
      times out too, purely from pool exhaustion.

WITH a bulkhead: recommendations_pool(30),
                  search_pool(70), separate.

t=0   recommendations starts hanging.
t=5   All 30 threads in recommendations_pool are
      blocked. search_pool's 70 threads are
      completely untouched.
t=6   The same burst of search requests arrives and
      is served normally from search_pool's full 70
      threads — recommendations' failure never
      reached search's capacity at all.

11: This is the cascading damage a shared pool allows — search requests fail purely because of contention with an unrelated dependency's outage.
21: The bulkheaded version shows the exact same recommendations outage causing zero impact on search — the isolation is structural, not just lucky timing.

Why this works: This is the concrete cost of skipping bulkheads — without them, ANY dependency's failure has the structural potential to take down every OTHER dependency's traffic too, purely through shared resource contention, regardless of how unrelated the two dependencies actually are.

Sizing a bulkhead pool without checking it can actually satisfy that dependency's own peak concurrency needs

Wrong

text

# arbitrary equal split across 4 dependencies,
# no regard for each one's real traffic pattern
pool_a = ThreadPool(size=25)
pool_b = ThreadPool(size=25)
pool_c = ThreadPool(size=25)
pool_d = ThreadPool(size=25)

Better

text

# sized from each dependency's own measured
# peak concurrency, summing to the total budget
pool_a = ThreadPool(size=50)  # high-traffic
pool_b = ThreadPool(size=10)  # low-traffic
pool_c = ThreadPool(size=30)  # medium-traffic
pool_d = ThreadPool(size=10)  # low-traffic

What you see: A dependency that legitimately needs more concurrent capacity than the other three combined gets throttled by its own bulkhead during completely normal peak traffic, even while the other three bulkheads sit almost entirely idle — the isolation "worked" but the sizing did not match reality.

Why: An equal split ignores that different dependencies have genuinely different traffic volumes and concurrency needs — a bulkhead sized purely for isolation without regard to real measured demand just relocates the capacity problem from "one shared pool, unpredictable contention" to "one specific pool, predictably too small," which is not actually progress for the dependency that got under-sized.

Bulkheaded thread pools, one per dependency

recommendations_pool (30)

blocked

search_pool (70)

healthy

One shared pool vs bulkheaded pools, same total capacity

One shared pool vs bulkheaded pools, same total capacity
Property	One shared pool (no bulkhead)	Bulkheaded (per-dependency pools)
Effect of dependency A hanging	Can consume the entire shared pool, starving calls to B, C, D	Bounded to A's own dedicated slice; B, C, D unaffected
Resource efficiency	Higher — no idle capacity sitting unused in another workload's slice	Lower — a quiet dependency's slice sits partially idle
Failure isolation	None — one dependency's failure can take down calls to every other dependency	Strong — failure is contained to the one dependency's own bulkhead

Remember: A bulkhead partitions a shared resource into dedicated, isolated slices per workload or dependency — named for a ship's watertight compartments — so one workload exhausting its own slice cannot also consume another workload's allocation. Size each slice from that dependency's real measured demand, and combine bulkheads with circuit breakers rather than choosing one over the other.

Combining circuit breakers, bulkheads, timeouts and bounded retries

standardintermediate

None of the four resilience techniques covered so far — timeouts, bounded retries, circuit breakers, bulkheads — substitutes for the others, because each answers a different question about the same failing call. A timeout answers "how long do I wait for one attempt." A bounded retry policy answers "how many times do I try before giving up." A circuit breaker answers "should I even attempt this call at all, given recent history." A bulkhead answers "how much of my own resources can this dependency ever consume, even while all of the above are still deciding." A resilient call to an external dependency typically uses all four together, layered around the same call.

Think of it as

Think of the four as separate safety systems on a car, not competing designs for the same one. A seatbelt (timeout) limits how far you can be thrown forward in a single sudden stop. Airbags (bounded retries) give you a limited number of cushioned "tries" at surviving an impact, not unlimited ones. A collision-avoidance system (circuit breaker) tries to stop you from getting into the crash at all once it has strong evidence one is coming. And crumple zones (bulkheads) contain the damage from spreading past the front of the car into the passenger compartment. A car with only one of these is much less safe than a car with all four working together, and none of them is a substitute for the others.

text

bulkhead(dependency_pool):
  if breaker.is_tripped(dependency):
    return fail_fast()
  for attempt in range(max_retries):
    try:
      return call(dependency, timeout=deadline)
    except TransientError:
      wait(backoff(attempt))
  breaker.record(failure)
  return give_up()

What we're doing: Show all four layers active in one call and which layer actually stops the eventual failure.

layered-defenses.txttext

Config: connect+read timeout 2s, max 3 retries with
        backoff, circuit breaker trips at 50% failure
        rate over 20 calls, dedicated bulkhead pool of
        15 threads for this dependency.

Dependency starts failing at t=0.

t=0-8s   Requests 1-20 each retry up to 3 times with
         backoff, each attempt bounded by the 2s
         timeout. Some threads in the 15-thread
         bulkhead are busy, but never more than 15 —
         other dependencies' pools are untouched.
t=8s     Failure rate over the last 20 calls crosses
         50% -> circuit breaker trips to Open.
t=8s+    New requests fail IMMEDIATELY, no timeout
         wait, no retries attempted at all — the
         breaker is now doing the work the timeout
         and retry logic were doing on every request
         up to this point.
t=38s    Cooldown elapses, breaker moves to Half-Open,
         sends exactly 1 trial call through the
         bulkhead's pool to check for recovery.

8: Before the breaker trips, the bulkhead is the only thing limiting how much damage the failing dependency can do to the rest of the service — timeouts and retries are actively running but not yet stopping the underlying pattern.
15: This is the moment the circuit breaker takes over as the primary defense — from here, timeouts and retries are no longer even attempted, which is a fundamentally different (and cheaper) kind of protection.

Why this works: This is the concrete handoff between the four layers over time — early on, timeouts/retries/bulkhead do the work of bounding cost per attempt and per dependency; once the pattern is clear, the circuit breaker takes over and removes the cost of attempting at all.

Retrying inside a circuit breaker's own trial call, defeating the point of a single half-open test

Wrong

text

if breaker.state == "half_open":
    for attempt in range(3):     # retries INSIDE
                                   # the trial call
        try:
            return call(dependency)
        except:
            continue
    breaker.record(failure)

Better

text

if breaker.state == "half_open":
    try:
        result = call(dependency)   # exactly ONE
                                      # attempt, no
                                      # retry loop
        breaker.record(success)
        return result
    except:
        breaker.record(failure)
        raise

What you see: A circuit breaker's half-open trial takes far longer to fail than expected, and the dependency being tested receives 3x the load the trial was supposed to represent — because the retry logic that normally wraps every call was left active inside the trial call too, turning "one careful test" into "the same unbounded attempt pattern that got us tripped in the first place."

Why: The half-open state's entire value is testing recovery with minimal, controlled exposure — wrapping that single trial in the normal retry policy reintroduces the multiplied load the breaker exists to prevent, right at the exact moment (a possibly-still-recovering dependency) where that load is most likely to cause a re-trip.

Four layered defenses, four different questions

Bulkhead

how much resource this can consume

Circuit breaker

whether to attempt at all

Bounded retry

how many attempts

Timeout

how long each attempt waits

What each technique answers, and what it does not cover alone

What each technique answers, and what it does not cover alone
Technique	Question it answers	What it does not cover alone
Timeout	How long to wait for one attempt	How many attempts, or whether to attempt at all
Bounded retry	How many attempts before giving up	Whether a whole dependency is known to be failing across requests
Circuit breaker	Whether to even attempt this call, given recent history	How much shared resource an in-flight attempt can consume
Bulkhead	How much of a shared resource this dependency can ever consume	When to actually stop attempting a specific failing call

Remember: Timeouts, bounded retries, circuit breakers and bulkheads answer four different questions about the same failing call and are meant to be layered together, not chosen between — a bulkhead bounds resource consumption, a circuit breaker decides whether to attempt at all, bounded retries govern how many attempts, and a timeout bounds each individual attempt.