beginnerintermediateadvanced

System Design · Section 65

High Availability

Level: intermediate
Read: 18 min
Concepts: 3

High availability starts with removing single points of failure across every layer a request passes through -- load balancer, application instances, database, cache, queue, storage, and networking -- since a system's real availability is capped at its least redundant layer, no matter how well the other layers were handled. Once each layer has its own redundant capacity, that capacity has to be arranged as either active-active (every node serves live traffic, so failover is instant but concurrent writes need conflict resolution) or active-passive (one node serves at a time, so failover means detecting the failure and promoting a standby, which takes real time but avoids concurrent-write conflicts entirely). Redundant capacity only helps during a real incident if something notices and reacts to a failure, though -- health checks detect it (only if they exercise a real dependency, not just process liveness), automatic failover reacts without waiting for a human, and graceful degradation covers whatever failover could not fully route around, so a partial failure degrades the user experience instead of becoming a hard outage.

This section

Design Patterns for High Availability: What gets you 99.999% uptime?Gaurav Sen

What is true here

A system is only as available as its least redundant layer -- eliminating SPOFs means checking all seven named layers explicitly, not just the ones that are easy to remember.
Active-active trades consistency complexity (resolving concurrent writes) for near-instant failover; active-passive trades failover speed (detect, then promote) for single-writer simplicity.
A health check has to exercise a real dependency to be useful -- one that only confirms the process is running can report healthy straight through an outage users are actively experiencing.
Automatic failover reacts to a detected failure without a human in the loop, but it is not infinite -- graceful degradation is what covers the case where there is nowhere healthy left to fail over to.

What you will be able to do

Audit a system layer by layer (LB, app instances, database, cache, queue, storage, network) and name every single point of failure explicitly
Choose between active-active and active-passive for a given component based on its cost, failover-speed, and consistency requirements
Write a health check that reflects real dependency health rather than just process liveness
Design a graceful-degradation fallback for the case where automatic failover has no healthy target left to route to

Building redundant capacity

Removing single points of failure across every layer, then choosing how that redundant capacity is arranged for failover.

Removing single points of failure across every layer

coreintermediate

A single point of failure (SPOF) is any one component whose failure alone takes the whole system down. High availability is not a property of one clever component — it is the result of walking every layer a request passes through (load balancer, application instances, database, cache, queue, storage, network) and making sure each one individually survives losing an instance, a zone, or a link. Missing even one layer means the system's real availability is capped at that one un-redundant component, no matter how redundant everything else is.

Think of it as

Think of the request path as a chain of links, each link being one layer of the stack. A chain is only as strong as its weakest link — it does not matter if six of seven links are forged from titanium if the seventh is a paperclip. Eliminating SPOFs means inspecting every link, not just the ones that are easy or obvious to make redundant (people naturally reach for redundant app servers first because it is the cheapest and most familiar move, then stop).

text

for each layer in [load_balancer, app_instances, database,
                    cache, queue, storage, network]:
    ask: "if exactly this one thing fails right now,
          does the system stay up?"
    if no: it is a SPOF -- add redundancy or accept the risk explicitly

overall availability ~= availability of the LEAST
redundant layer in the path

What we're doing: Audit a system that already calls itself "highly available" and find the layers that were skipped.

spof-audit.txttext

Claimed: "highly available" -- 4 app instances,
autoscaling enabled.

Layer-by-layer audit:
  Load balancer   -> 1 instance, single AZ         SPOF
  App instances   -> 4 instances, 2 AZs            OK
  Database        -> 1 primary, 1 replica, 2 AZs   OK
  Cache            -> 1 Redis node, no replica      SPOF
  Queue            -> 3-node cluster, replicated    OK
  Storage          -> single-zone bucket            SPOF
  Network          -> redundant paths, multi-AZ     OK

Three SPOFs found despite the "highly available"
claim: load balancer, cache, storage. Any one of
them failing takes down the whole system regardless
of how well app instances, database, queue and
network were handled.

7: A single load balancer instance undoes every other layer's redundancy -- nothing can reach the four redundant app instances if it goes down.
10: A single Redis node with no replica is a SPOF even though the queue right below it was clustered correctly -- each layer has to be checked on its own.
12: A single-zone bucket is a SPOF for the same reason a single-AZ database would be: losing that one zone loses the data path entirely.

Why this works: The audit shows why "we have multiple app instances" is not the same claim as "we removed single points of failure" -- each of the seven layers can independently hide an un-redundant component, and app instances happening to be redundant says nothing about the other six.

Treating "we autoscale the app tier" as equivalent to "no single points of failure"

Wrong

text

# Autoscaling group covers app instances,
# nothing else gets checked
app_instances: min=2, max=10, autoscaling=true
load_balancer: 1 instance   # never revisited
cache: 1 node               # never revisited
storage: single-zone bucket # never revisited

Better

text

# Every layer reviewed independently, not just
# the one that autoscaling made easy
app_instances: min=2, max=10, autoscaling=true, 2 AZs
load_balancer: redundant pair, floating IP
cache: clustered, 3 nodes, replicated
storage: multi-zone bucket

What you see: The app tier survives instance loss cleanly in every drill, then the entire system goes down the first time the single cache node or single-zone bucket has an incident -- a layer nobody thought to re-check because the autoscaling group made the app tier feel "handled."

Why: Autoscaling only guarantees the app tier keeps its instance count -- it says nothing about the load balancer, cache, queue, storage, or database sitting next to it. Each layer needs its own explicit redundancy decision; solving one layer well creates a false sense that the whole system is covered.

Every layer on the request path needs its own redundancy

Load balancer

redundant pair, not one instance

Application instances

spread across multiple AZs

Database

primary + replica(s), automated failover

Cache

clustered, not one node

Queue / broker

replicated partitions

Storage

replicated disks or multi-zone object store

Network

redundant paths, no single switch or NIC

Layer-by-layer SPOF elimination across the request path

Layer-by-layer SPOF elimination across the request path
Layer	Common SPOF	Redundancy technique
Load balancer	One LB instance or one AZ hosting it	Redundant LB pair (active-active or active-passive) with a floating/virtual IP or DNS-based failover
Application instances	Too few instances, or all in one AZ	Multiple stateless instances spread across at least two availability zones
Database	Single primary with no replica	Primary plus one or more replicas, with automated or manual failover promotion
Cache	Single cache node holding all keys	Clustered cache (e.g. replicated or sharded) so one node's loss does not evict the whole cache
Queue / broker	Single broker node or single partition leader	Clustered broker with replicated partitions and leader election on node loss
Storage	Single disk or single-zone bucket	Replicated block storage (e.g. RAID or a distributed filesystem) or a multi-zone object store
Networking	Single NIC, switch, or availability zone network path	Redundant network paths and multi-AZ deployment so no single network element is load-bearing

Remember: High availability means walking all seven layers explicitly -- load balancer, application instances, database, cache, queue, storage, network -- and giving each one its own redundancy answer. Fixing the layers that are easy to remember while skipping cache, queue, storage or networking leaves a real SPOF hiding behind a system that otherwise looks redundant.

Active-active vs active-passive

coreintermediate

Active-active runs two or more redundant instances (servers, databases, or whole regions) that all serve live traffic at the same time -- if one fails, the others simply absorb its share of load, with no failover step needed. Active-passive keeps one instance serving traffic while one or more standby instances sit idle (or replicating data) until the active one fails, at which point a failover promotes a standby to active. The choice trades cost and consistency complexity against failover speed and wasted capacity.

Think of it as

Active-active is like two cashiers both ringing up customers at once -- if one cashier steps away, the line simply funnels to the other with no announcement needed, but the store has to keep both registers' cash drawers in sync in real time. Active-passive is like a cashier with a backup sitting in the break room -- cheaper to keep one register idle than staff two at once, but when the active cashier leaves, someone has to notice, walk to the break room, and get the backup up to the register before the line can move again.

text

active-active:
  node_a.serves(traffic)  # both serving now
  node_b.serves(traffic)
  # node_a fails -> node_b already absorbing load, no promotion step

active-passive:
  active.serves(traffic)
  standby.replicate(active)  # idle for traffic, staying in sync
  # active fails -> detect failure -> promote(standby) -> standby.serves(traffic)

What we're doing: Compare how the same database failure plays out under each topology.

topology-failure.txttext

Active-passive database:
  t=0    primary serving writes, replica in sync
  t=5s   primary crashes
  t=8s   health check detects primary is down
  t=15s  replica promoted to primary
  t=15s  writes resume -- 15s of write unavailability

Active-active database (multi-master, 2 nodes):
  t=0    both nodes accepting writes
  t=5s   node A crashes
  t=5s   node B was already accepting writes --
         traffic pinned to A reroutes to B
  t=5s   writes to B continue uninterrupted
  # but: any write that was in flight to A and not
  # yet replicated to B needs conflict resolution
  # once A rejoins

4: The gap between crash and detection is unavoidable overhead in active-passive -- nothing can promote a standby before the failure is even noticed.
6: The promotion step itself takes real time -- a standby has to become writable, which is not instantaneous even once the failure is confirmed.
13: Active-active avoids the promotion delay entirely, but trades it for a genuinely hard problem: reconciling writes that were in flight to the failed node when it went down.

Why this works: The same failure produces a 15-second write gap under active-passive and effectively no write gap under active-active -- but active-active only avoided that gap by taking on a conflict-resolution problem that active-passive never has to solve at all.

Choosing active-active for a system that cannot actually resolve write conflicts

Wrong

text

# Two "active" database nodes, both accepting
# writes, with no conflict resolution strategy
node_a.accept_writes()
node_b.accept_writes()
# assume replication just "figures it out"

Better

text

# Either use active-passive (one writer, no
# conflicts possible), or active-active with an
# explicit conflict-resolution strategy
# (e.g. last-write-wins, CRDTs, or partition
# writes by key so each key has one owner)
if not have_conflict_resolution_strategy():
    use_active_passive()  # simpler, safer default

What you see: Two customers update the same record within milliseconds of each other on different nodes; both writes "succeed" locally, and depending on replication order one silently overwrites the other with no error or warning -- the system loses a write and nobody notices until a customer reports data that reverted.

Why: Active-active only removes failover delay if every node can safely accept writes at the same time, which requires an explicit answer to "what happens when two nodes get conflicting writes for the same data." Adopting active-active for its failover-speed benefit while skipping that harder question just trades a visible outage for a silent data-correctness bug -- often the worse failure mode.

Active-active vs active-passive on failure

Active-active

+Both nodes serve live traffic simultaneously
+One node fails -> the other already has capacity, no promotion needed
+Requires resolving concurrent writes across nodes

Active-passive

−Only the active node serves traffic; standby stays idle or replicating
−Active fails -> detect failure, then promote standby -> traffic resumes
−Only one writer at a time, so no concurrent-write conflicts

Active-active vs active-passive trade-offs

Active-active vs active-passive trade-offs
Dimension	Active-active	Active-passive
Cost / resource utilization	Higher -- all nodes run live capacity, none sits idle	Lower -- standby capacity sits mostly idle until failover
Failover speed	Effectively instant -- no promotion step, traffic just redistributes	Slower -- failure must be detected, then a standby promoted before it can serve
Data-consistency complexity	High -- concurrent writes to multiple nodes need conflict resolution or partitioning	Low -- one writer at a time, so no concurrent-write conflicts to resolve
Best fit	Stateless services, read-heavy data tiers, systems that can tolerate eventual consistency	Single-writer systems (most relational databases), or anything where correctness matters more than failover speed

Remember: Active-active removes failover delay by keeping every node serving live traffic, at the cost of having to resolve concurrent writes across nodes. Active-passive avoids that consistency problem by keeping one writer at a time, at the cost of a failover gap while a standby gets detected and promoted. Neither is universally better -- match the topology to whether the component can safely accept concurrent writes at all.

Making redundancy work during a real failure

Health checks, automatic failover, and graceful degradation as the mechanics that turn redundant capacity into an actual recovery.

Health checks, automatic failover and graceful degradation working together

coreintermediate

Redundant capacity only helps during a real failure if something notices the failure and reacts to it -- that is the job of these three mechanics working in sequence. Health checks continuously ask "is this component still working?" so a failure gets detected instead of assumed. Automatic failover reacts to a failed health check by rerouting traffic or promoting a standby without waiting for a human. Graceful degradation is the fallback for the failures automatic failover cannot fully hide -- serving a reduced but still-useful response (cached data, a simplified feature set) instead of a hard error when full functionality genuinely is not available.

Think of it as

Think of a building's fire safety system as three layered mechanics. Smoke detectors are the health check -- they continuously watch for a problem and raise an alarm the instant one is found, rather than waiting for someone to notice smoke by smell. The sprinkler system is automatic failover -- it reacts to the detected problem immediately, with no human in the loop, because waiting for a person to arrive and act would be too slow. And the building's emergency lighting and marked exits are graceful degradation -- when the fire has actually taken out normal power, the building does not go completely dark and unusable, it falls back to a reduced-but-functional state that still gets people out safely.

text

loop every N seconds:
    result = health_check(component)   # detect
    if result == unhealthy:
        automatic_failover(component)  # react without a human

# when even failover cannot restore full service:
def handle_request():
    try:
        return full_response()
    except DependencyUnavailable:
        return degraded_response()  # cached / reduced, not a hard error

What we're doing: Trace one dependency outage through all three mechanics in sequence.

ha-mechanics-trace.txttext

1. Recommendation service's health check starts
   failing -- it responds to /health but every real
   request now times out (a shallow check would
   have missed this).
2. The check is deep enough to hit a lightweight
   real query, so it correctly reports "unhealthy"
   after 3 consecutive failures.
3. Automatic failover reroutes recommendation
   traffic to a healthy replica in another zone --
   no human paged yet.
4. The replica is also overloaded by the sudden
   full traffic shift and starts timing out too --
   failover has nowhere else to send traffic.
5. The product page's graceful degradation kicks
   in: it renders without personalized
   recommendations (a generic "popular items" list
   instead), rather than failing the whole page.
6. Users see a slightly less personalized page.
   Checkout, search, and every other feature keep
   working normally.

3: A shallow health check (just "is the process up") would have reported healthy while every real request failed -- the check has to exercise the same path a real request does.
10: Failover is not infinite -- once there is no healthy target left to route to, it cannot manufacture capacity that does not exist.
13: This is the step that turns a dependency outage into a minor visual downgrade instead of a broken checkout page -- graceful degradation is what covers the gap failover could not close.

Why this works: None of the three mechanics alone gets from "a service is failing" to "users barely notice" -- detection without automatic reaction just produces an alert nobody acts on fast enough, and automatic failover without a degradation fallback still fails hard the moment there is nowhere left to fail over to.

Building automatic failover but treating "no healthy target left" as an unhandled case

Wrong

text

def get_recommendations():
    for instance in healthy_instances():
        return instance.call()
    # falls through with no return if the list
    # is empty -- raises an unhandled exception
    # that takes down the whole product page

Better

text

def get_recommendations():
    for instance in healthy_instances():
        return instance.call()
    return popular_items_fallback()  # degrade,
    # don't propagate the failure to the whole page

What you see: Failover works perfectly for a single-instance failure, but the first time an entire dependency is degraded (every instance unhealthy at once, e.g. during a regional issue) the unhandled case throws all the way up and takes down the whole page instead of just the personalization widget -- because the "no target left" branch was never written, only assumed unreachable.

Why: Automatic failover is built to handle "this one instance is down," and it is easy to stop there because that is the common case seen in testing. The rarer case -- every redundant target is unhealthy at once -- is exactly when graceful degradation is supposed to take over, but only if that fallback path was actually written rather than left as an implicit assumption that failover always has somewhere to send traffic.

Detection, reaction, and fallback in sequence

Health check

is it really serving requests?

Failure detected

Automatic failover

reroute or promote, no human

Service restored

Graceful degradation

reduced response, not a hard error

The three mechanics and what each is responsible for

The three mechanics and what each is responsible for
Mechanic	Question it answers	What happens if it is missing
Health checks	Is this component actually able to serve a real request right now?	A dead component keeps receiving traffic because nothing noticed it failed
Automatic failover	Now that a failure is detected, how fast can traffic move away from it without a human?	Detection happens, but recovery waits on a human to notice the alert and act -- minutes to hours of extra downtime
Graceful degradation	When failover cannot fully restore normal service, what is the best partial response?	Any failure failover cannot fully route around becomes a hard error for every user, instead of a reduced-but-working response

Remember: Health checks detect a real failure (not just "is the process up"), automatic failover reacts to that detection without waiting for a human, and graceful degradation is the fallback for whatever failover could not fully route around. Skip the first and failover never triggers; skip the last and any failure failover cannot fully absorb becomes a hard outage instead of a reduced-but-working response.