High Availability

Key features

• A SPOF is defined by blast radius, not by rarity — a component that fails once a decade but takes the whole system down when it does is still a SPOF
• Each of the seven layers needs its own redundancy technique — there is no single fix that covers load balancer, database, and network all at once
• Databases split into two different SPOF problems: losing the primary (needs a replica that can be promoted) and losing a whole zone or region (needs cross-zone/region replication)
• Networking SPOFs are the easiest to forget — a single NIC, a single top-of-rack switch, or a single availability zone's network path can undo redundancy at every other layer
• Removing a SPOF always costs something — more instances, cross-zone replication lag, or operational complexity — so the checklist is a prioritization tool, not an obligation to make every layer maximally redundant everywhere

Command

for each layer in [load_balancer, app_instances, database,
                    cache, queue, storage, network]:
    ask: "if exactly this one thing fails right now,
          does the system stay up?"
    if no: it is a SPOF -- add redundancy or accept the risk explicitly

overall availability ~= availability of the LEAST
redundant layer in the path

Key features

• Active-active has no failover delay for reads or writes routed to a surviving node -- capacity that was already serving traffic simply absorbs more
• Active-passive wastes standby capacity between failovers, but sidesteps the hardest problem in active-active: keeping multiple simultaneously-written copies of data consistent
• Active-active for a database usually means either read replicas serving reads (writes still go to one primary) or a genuinely multi-master setup that must resolve write conflicts
• Active-passive failover is not instant -- it requires detecting the failure, then promoting a standby, and that promotion window is exactly the availability gap active-active avoids
• The choice is rarely all-or-nothing across a whole system -- one component can run active-active (stateless app servers) while another right next to it runs active-passive (a single-writer database)

Command

active-active:
  node_a.serves(traffic)  # both serving now
  node_b.serves(traffic)
  # node_a fails -> node_b already absorbing load, no promotion step

active-passive:
  active.serves(traffic)
  standby.replicate(active)  # idle for traffic, staying in sync
  # active fails -> detect failure -> promote(standby) -> standby.serves(traffic)

Key features

• A health check is only useful if its failure definition matches real user impact -- a check that only pings a process (not its ability to serve a real request) can report "healthy" while users see errors
• Automatic failover needs the health check to be right consistently, not just once -- a single flaky check triggering failover causes unnecessary disruption (a false positive failover is itself an outage)
• Graceful degradation is what runs when failover cannot fully restore capacity -- e.g. a downstream dependency is down and there is no redundant copy of it to fail over to at all
• These three mechanics form a chain, not independent options: health checks detect, failover reacts, and degradation covers whatever failover could not fix
• A system with automatic failover but no graceful degradation still fails hard the moment something breaks that failover cannot route around -- degradation is what keeps a partial failure from becoming a total outage for the end user

Command

loop every N seconds:
    result = health_check(component)   # detect
    if result == unhealthy:
        automatic_failover(component)  # react without a human

# when even failover cannot restore full service:
def handle_request():
    try:
        return full_response()
    except DependencyUnavailable:
        return degraded_response()  # cached / reduced, not a hard error