Availability, Reliability and Durability

What we're doing: Show a single incident scenario touching all three guarantees differently, to make the distinction concrete.

A database node crashes during a write.

  Availability: the service returns errors for 90 seconds
                while a replica is promoted -> availability hit

  Reliability:  during those 90 seconds, some in-flight
                requests silently returned stale reads before
                erroring -> reliability hit

  Durability:   the write itself was already replicated to
                two other nodes before the crash, so no data
                was lost -> durability held

3

The 90-second outage is purely an availability number — the service could not be reached.

6

Stale reads during the failover are a reliability problem — the system was reachable but gave wrong answers.

10

Because the write was already replicated, no committed data was lost — durability held even though availability and reliability both took a hit.

Why this works: The same incident can score very differently on each axis. Conflating them leads to the wrong fix — adding more replicas improves durability, not necessarily availability during a failover, and does nothing for a reliability bug in the application code.

What we're doing: Compute the downtime budget for a real target and show how little room 99.99% actually leaves.

SLA target: 99.99% availability.

  Yearly budget:  525,600 min × 0.0001  ≈ 52.6 minutes
  Monthly budget: 43,800 min × 0.0001   ≈ 4.4 minutes
  Weekly budget:  10,080 min × 0.0001   ≈ 1.0 minute

One unplanned 10-minute outage uses about 19% of
the ENTIRE year's downtime budget.

3

The yearly budget converts directly from the availability percentage and the number of minutes in a year.

8

A single 10-minute outage against a 52.6-minute yearly budget is a large fraction of the entire year's allowance — 99.99% leaves very little room for even one incident.

Why this works: Availability targets sound similar as percentages (99% vs 99.99% looks like a small gap) but translate to wildly different operational realities — the difference between "restart it by hand next business day" and "no human can be in the loop."

What we're doing: Show all seven techniques applied together in one incident, and what each one specifically prevented.

An availability zone loses power.

  Redundancy:     app servers and DB replicas exist in 2 other zones
  Health checks:  load balancer detects the zone's servers stop
                  responding within 10 seconds
  Failover:       traffic reroutes to the healthy zones automatically
  Graceful
  degradation:    the recommendations service (in the failed zone)
                  is skipped; the rest of the page still renders
  Replication:    the DB replica in a healthy zone is promoted,
                  no committed writes are lost
  Backups:        unrelated — used later to confirm no silent
                  corruption occurred during the failover
  Disaster
  recovery plan:  not invoked — a single zone loss was absorbed
                  by redundancy; DR is reserved for a full region loss

3

Redundancy across zones is what makes any of the rest of this possible.

5

Health checks are the detection mechanism — without them, failover has nothing to trigger on.

9

Graceful degradation keeps the page usable even though one non-critical dependency is down.

12

Replication with a promotable replica is what prevents this from becoming a durability incident, not just an availability one.

17

Disaster recovery is reserved for a scale of failure redundancy alone cannot absorb — this incident never reached that threshold.

Why this works: These seven techniques are not interchangeable — each defends a distinct failure mode, and a real incident usually exercises several of them together. Naming which technique handled which part of an incident is what makes a postmortem specific instead of vague.

What we're doing: Walk a request path end to end and find the single point of failure hiding behind an otherwise-redundant design.

Claimed: "highly available — 3 app servers,
load-balanced."

Walk the full request path:
  DNS             -> managed, multi-region        OK
  Load balancer   -> ONE instance, no standby      SPOF
  App servers     -> 3 instances, redundant        OK
  Database        -> 1 primary, no replica          SPOF
  Object storage  -> managed, multi-zone           OK

Two single points of failure found despite the
"highly available" claim: the load balancer and
the database. Either one failing takes the whole
system down regardless of the redundant app tier.

7

A single load balancer instance is a SPOF even though everything behind it is redundant — nothing can reach the redundant servers if it goes down.

9

A single database primary with no replica is a SPOF for both availability and durability — its failure is total, not partial.

12

The original claim only looked at the app tier. Walking the entire path surfaced two SPOFs it missed.

Why this works: Redundancy at one layer creates a false sense of safety if another layer is not checked. The system's real availability is bounded by its single least redundant component, not by whichever layer got the most attention.