beginnerintermediateadvanced

System Design · Section 64

Disaster Recovery

Level: intermediate
Read: 15 min
Concepts: 3

Disaster recovery is a toolkit, not one technique: backups and point-in-time recovery undo logical failures (corruption, bad deploys, accidental deletes) that replication cannot, because replication (§22) faithfully copies bad writes just as fast as good ones — it protects against hardware and instance failure instead. Multi-zone deployment survives a single data-center failure; multi-region strategies survive a whole-region outage, at real added cost and complexity. A DR strategy only becomes a testable guarantee once it states two independent numbers: RPO (the maximum acceptable data loss, set by backup/replication cadence) and RTO (the maximum acceptable downtime, set by how automated and rehearsed recovery actually is) — a strong RPO does not imply a strong RTO, or vice versa. None of it means anything, though, until it is actually drilled: a backup that reports success every night for years and has never been restored is not evidence it can rebuild a working system, and a standby that has never taken real traffic is an untested hypothesis about failover, not a capability.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Backups + PITR undo logical failures; replication (§22) recovers from hardware failure — neither substitutes for the other.
Multi-zone survives one data-center failure; multi-region survives a whole-region outage, at meaningfully higher cost.
RPO (max data loss) and RTO (max downtime) are independent numbers — tightening one does not tighten the other.
A backup job reporting success only proves bytes were written, never that the data can rebuild a working system.
Restore and failover drills need a recurring schedule, triggered again after schema or infrastructure changes.

What you will be able to do

Pick the right DR mechanism (backup/PITR, replication, multi-zone, multi-region) for a given failure, not just "have some DR"
State RPO and RTO for a system explicitly, and check the current toolkit can actually achieve both numbers
Explain why replication does not substitute for backups, and why a green backup dashboard does not substitute for a restore drill
Design a recurring restore/failover drill schedule instead of relying on a single past successful test

The toolkit and its guarantee

What each DR mechanism actually protects against, and the two numbers — RPO and RTO — that turn a combination of them into a testable guarantee.

The DR toolkit: backups, point-in-time recovery, and multi-zone/multi-region deployment

coreintermediate

Disaster recovery is not one technique — it is a small toolkit of mechanisms, each protecting against a different failure blast radius, and a real DR strategy usually combines several. Backups are a copy of data taken at a point in time and stored somewhere independent of the live system, so a bad deploy, a bug that corrupts rows, or outright deletion can be undone by restoring from before it happened. Point-in-time recovery (PITR) is a refinement of backups: instead of only being able to restore to the moment of the last full snapshot, a continuous log of changes (write-ahead log, binlog, oplog) lets you restore to any specific second — critical when the disaster is "we discovered the corruption six hours after it happened" rather than "the whole server died." Replication (covered in depth in §22 — primary/replica, sync vs. async) keeps a second live copy up to date in near-real-time, protecting against hardware failure but not against corruption or deletion that replicates just as faithfully as good data. Multi-zone deployment spreads instances across a cloud provider's availability zones (independent power, cooling, and networking within one region) so a single data-center-level failure does not take the whole system down. Multi-region strategies go further, keeping a fully separate copy of the system in a geographically distant region, protecting against a failure that takes out an entire region — the rarest and most expensive failure to defend against, reserved for the highest-stakes systems.

Think of it as

Think of it as concentric rings of protection, each guarding against a bigger blast radius than the last. Backups and PITR are the innermost ring — like a "restore previous version" button on a document, protecting against your own mistakes (bad data, bad deploys, accidental deletes). Replication is the next ring out — a hot spare copy of the machine itself, protecting against one server dying, but it will faithfully copy a corrupted document too, which is why it does not replace backups. Multi-zone is the next ring — like having that hot spare in a different building on the same campus, so a fire in one building doesn't take out both copies. Multi-region is the outermost, most expensive ring — a hot spare in a different city entirely, for the disaster where the whole campus is unreachable. Nobody needs every ring for every system; the point of DR planning is choosing how many rings a given system's failure cost actually justifies.

text

// A layered DR posture, cheapest ring first
1. Backups (daily full + continuous change log for PITR)
2. Replication (§22): sync within a region, async cross-region
3. Multi-zone: instances + replicas spread across 3 AZs
4. Multi-region: standby copy in a second region, promoted on
   a declared regional disaster

What we're doing: Trace which DR mechanism actually recovers a system from three different failures.

dr-toolkit-scenarios.txttext

Failure A: A bad migration silently corrupts a
column across 40% of rows; discovered 5 hours later.
  -> Replication already copied the corruption to
     every replica. Only point-in-time recovery
     (restore to 5 hours ago) fixes this.

Failure B: The primary database instance's disk
fails hard, mid-afternoon.
  -> A replica (§22) is promoted to primary in
     seconds to minutes. Backups are not needed for
     this failure at all -- it's not a data problem.

Failure C: The entire availability zone hosting the
primary and its replica loses power.
  -> If both live in the same zone, Failure B's
     replica is gone too. Multi-zone deployment
     (replicas spread across zones) is what
     actually survives this one.

6: Point-in-time recovery is the only tool here — replication actively worked against recovery by copying the bad data everywhere, faster than a human noticed.
11: This is exactly what replication is for — fast failover to a hot copy, no backup restore needed or wanted.
17: A DR plan that stopped at "we have replicas" fails here if those replicas share a failure domain with the primary — multi-zone spreads them so one zone-level event cannot take out both.

Why this works: The three failures need three different tools from the same toolkit — showing why "we have backups" or "we have replication" alone is never a complete answer; the failure determines which ring of protection actually recovers the system.

Treating replication as a substitute for backups

Wrong

text

# DR plan, in full:
# "We have a synchronously replicated standby,
#  so we're covered."
# No backup schedule, no PITR change log retained.

Better

text

# DR plan:
# - Replication (§22): sync standby for fast
#   failover on instance/hardware failure
# - Daily full backups + continuously retained
#   WAL/binlog for point-in-time recovery, kept
#   in a separate storage account/region
# - Backups protect against corruption and human
#   error; replication protects against hardware
#   failure. Neither substitutes for the other.

What you see: A bad deploy corrupts data at 2pm; by 2:00:03pm the synchronous replica has faithfully copied the exact same corruption. The team discovers the plan's "disaster recovery" was only ever hardware-failure recovery — there is no snapshot or change log to restore from, so the corrupted state is now the only state that exists anywhere.

Why: Replication's entire value is copying changes quickly and faithfully — which is precisely why it cannot distinguish a good write from a bad one. Backups (with PITR) are the only mechanism in the toolkit that preserves a prior, pre-corruption state on purpose; skipping them because "we already replicate" conflates two tools that protect against different failure classes.

DR toolkit as concentric rings of protection

Backups + PITR

restore to any second, protects against logical failures

Replication (§22)

hot copy, protects against instance/hardware failure

Multi-zone deployment

survives one data-center failure

Multi-region strategy

survives a whole-region outage

The DR toolkit — what each mechanism actually protects against

The DR toolkit — what each mechanism actually protects against
Mechanism	Protects against	Does NOT protect against
Backups	Data corruption, bad deploys, accidental deletes	Being stale between backup intervals
Point-in-time recovery	The above, restorable to an arbitrary second	Requires a continuous change log, not just snapshots
Replication (§22)	Single-instance/hardware failure, fast failover	Corruption or deletion, which replicates faithfully too
Multi-zone deployment	One data center/availability zone going down	A failure affecting the whole region
Multi-region strategy	An entire region becoming unreachable	Cost, latency, and consistency trade-offs it introduces

Remember: The DR toolkit is backups + point-in-time recovery (logical failures — corruption, bad deploys, deletion), replication (§22 — hardware/instance failure, fast failover, but it replicates bad data just as well as good data), multi-zone (one data-center failure), and multi-region (a whole-region outage). None of these substitutes for another — they protect against different failure classes, and a real DR strategy picks a combination sized to what each system's downtime and data loss actually cost.

RPO vs. RTO: the two numbers that define a DR strategy's guarantee

coreintermediate

RPO (Recovery Point Objective) and RTO (Recovery Time Objective) are the two numbers that turn "we have disaster recovery" into an actual, testable guarantee — without them, "disaster recovery" is just a feeling. RPO answers "how much data can we afford to lose?", expressed as a time window: an RPO of 1 hour means that after recovering from a disaster, the restored data can be missing at most the last hour's worth of writes. It is driven entirely by how the toolkit's backup/replication cadence works — continuous synchronous replication can approach an RPO of zero, nightly backups imply an RPO of up to 24 hours. RTO (Recovery Time Objective) answers a different question — "how long can the system be down?" — measured from the moment the disaster is declared to the moment the system is back up and serving traffic again. A system can have an excellent RPO (almost no data lost) and a terrible RTO (it takes six hours to actually bring the restored system back online), or vice versa — they are independent numbers, driven by different mechanisms, and a DR strategy is only complete when both are stated and both are actually achievable by the toolkit in place.

Think of it as

Picture a company's filing cabinet catching fire. RPO is "how recent is the copy in the off-site safe?" — if the last copy made it into the safe five minutes before the fire, RPO is five minutes; if the safe is only restocked once a week, RPO is up to a week, and everything filed since the last restock is gone regardless of how fast the fire department arrives. RTO is a completely separate question: "how long until a working filing cabinet, with that safe copy in it, is back in the office and usable?" — that depends on how fast someone can drive to the safe, print everything out, and set the cabinet back up, which has nothing to do with how recent the copy was. A five-minute-old copy that takes eight hours to restore into a working cabinet has a great RPO and a bad RTO; a week-old copy that's back in place in ten minutes has a bad RPO and a great RTO. Both numbers matter, and neither one implies the other.

text

// Stating a DR requirement means stating both numbers
service: payments-ledger
RPO: 0        // synchronous replication, zero data loss tolerated
RTO: 5 min    // hot standby, automated failover already tested

service: internal-wiki
RPO: 24h      // nightly backup is acceptable
RTO: 4h       // manual restore from backup is acceptable

What we're doing: Compute RPO and RTO for one incident from a timeline of timestamps, and show they are genuinely independent.

rpo-rto-worked-example.txttext

09:58 - Last successful transaction replicated
        to the standby database.
10:00 - Primary region goes down (disaster).
10:02 - On-call engineer confirms it's a real
        regional outage, declares a disaster.
10:07 - Standby promoted to primary; DNS/traffic
        cutover begins.
10:15 - System fully serving traffic again from
        the standby region.

RPO = 10:00 - 09:58 = 2 minutes of data
      (any writes between 09:58 and 10:00 are lost)

RTO = 10:15 - 10:00 = 15 minutes of downtime
      (measured from disaster to full recovery,
       not from when it was declared)

10: RPO is purely about the data gap -- it was already "locked in" at 09:58, two minutes before anyone even knew there was a disaster.
13: RTO is measured from the disaster itself (10:00), not from the 10:02 declaration -- the 2 minutes it took to notice and confirm count against RTO too.

Why this works: The two numbers come from two different parts of the timeline and two different parts of the toolkit: RPO was fixed the moment replication last synced, well before the disaster; RTO depends on everything that happens after, including detection time, which is easy to forget belongs to RTO rather than being "free."

Measuring RTO from when the disaster was declared instead of when it happened

Wrong

text

# Incident report:
# "Disaster declared at 10:02, system back up at
#  10:15 -- RTO was 13 minutes, well within our
#  15-minute target."
# (Ignores the 2 minutes between 10:00 and 10:02
#  where the system was already down.)

Better

text

# Incident report:
# "Disaster occurred at 10:00, system back up at
#  10:15 -- actual RTO was 15 minutes."
# Detection and declaration time is real downtime
# and belongs inside the RTO measurement, not
# excluded from it.

What you see: RTO targets look consistently met in postmortems, yet users experienced longer outages than the reports admit — because every incident quietly starts the clock at "declared," not "occurred," shaving off however many minutes detection and triage actually took.

Why: RTO is meant to describe the user-facing downtime, and users do not care whether the system was down-and-known-about or down-and-not-yet-noticed — both are the system being unavailable. Starting the clock at declaration flatters the metric but hides exactly the detection-speed problem a DR drill is supposed to surface.

RPO and RTO on the same incident timeline

09:58
Last good backup/replica sync
start of the RPO window
10:00
Disaster occurs
data since last sync is the RPO loss
10:02
Disaster declared
detection time still counts toward RTO
10:15
System back up
end of the RTO window

RPO and RTO — what each measures and what tightens it

RPO and RTO — what each measures and what tightens it
Metric	Question it answers	What tightens it
RPO	How much data can we lose?	More frequent backups; synchronous replication; continuous PITR log retention
RTO	How long can we be down?	Automated failover/runbooks; a warm or hot standby already provisioned; rehearsed restore drills

Remember: RPO = how much data you can afford to lose (backward-looking, set by backup/replication cadence). RTO = how long you can afford to be down (forward-looking, set by how automated and rehearsed recovery is). They are independent numbers driven by different parts of the toolkit — tightening one does not tighten the other, and either number is worthless if it was written down without checking the actual infrastructure can hit it.

Proving it actually works

Why a backup or a standby that has never been exercised is an assumption, not a capability — and the drilling discipline that closes that gap.

Testing backups and failover: an untested backup is not evidence of recoverability

coreintermediate

A backup job that reports "success" every night for two years tells you a file was written somewhere — it tells you nothing about whether that file can actually rebuild a working system. Backups fail silently in ways a completion status never surfaces: the backup format drifts out of sync with a schema migration, credentials to the restore target quietly expire, a backup completes but is missing a table that was added after the backup script was last touched, or the backup is technically valid but restoring it takes eleven hours against a four-hour RTO. None of these show up as a red X in a dashboard — the job exits 0 every time, right up until the moment someone actually needs the data back and discovers, mid-incident, that the thing they were relying on does not work. The same logic applies to failover: a standby database or a secondary region that has never actually taken production traffic is a hypothesis, not a capability, because the failover path itself (DNS cutover, connection string changes, cache warm-up, dependent services reconnecting) has its own failure modes that only show up when it is actually exercised.

Think of it as

A backup you have never restored is like a fire extinguisher you have never test-fired — it hangs on the wall looking ready, the inspection sticker says it was checked, and the first time anyone finds out it does not work is the fire. A restore drill is pulling the pin and firing it into a bucket once a quarter specifically so that "looks fine" gets replaced with "confirmed fine." Same logic for failover: a generator that automatically kicks in during a power outage is only trustworthy if it has been started under load before, not just serviced on a checklist — the actual test is cutting the power and watching the lights, not reading a maintenance log that says the generator exists.

text

// A restore/failover drill, scheduled like any other test
quarterly_dr_drill:
  1. Pick last night's backup (not a golden known-good one)
  2. Restore it into an isolated environment
  3. Run schema/row-count/smoke-test checks against it
  4. Measure restore time against the stated RTO
  5. Promote a standby and cut traffic over in staging
  6. Record pass/fail + timing; file a ticket for any gap

What we're doing: Trace a real-shaped incident where an untested backup is discovered broken during the disaster itself, not before.

untested-backup-incident.txttext

Month 1-18: Nightly backup job runs, reports
"SUCCESS", dashboard is green every single day.
No one has ever restored one of these backups.

Month 14: A migration adds a new "refunds" table.
The backup script's table list was never updated
to include it -- it keeps backing up everything
it already knew about, still exits 0.

Month 19: Primary database is destroyed by a bad
storage-array firmware update.
  10:00 - Team pulls last night's backup to restore.
  10:45 - Restore completes. Dashboard was right --
          the backup itself was structurally valid.
  10:46 - Someone notices the "refunds" table is
          empty. It has been silently excluded from
          every backup for 5 months.
  10:47 - The only copy of 5 months of refund
          records no longer exists anywhere.

8: This is the moment the backup became incomplete -- and nothing about the job's "SUCCESS" status changed, because the job genuinely did succeed at backing up everything it was configured to include.
17: A restore drill run any time in those 5 months, with an actual row-count or schema check against the live system, would have caught this in minutes instead of during a real disaster.

Why this works: The backup job was never lying — "SUCCESS" was true the entire time, for a narrower definition of success than anyone realized. Only an actual restore, compared against the live system, would have surfaced the gap; a green dashboard measures "did the job run," not "can this data rebuild the system."

Verifying backups by checking the job status instead of by restoring

Wrong

text

# "DR verification" process, in full:
def verify_backups():
    return last_backup_job.status == "SUCCESS"
# Runs daily. Has returned True for 18 months.

Better

text

def verify_backups():
    restored = restore_to_isolated_env(last_backup)
    checks = [
        schema_matches_production(restored),
        row_counts_within_tolerance(restored, prod),
        restore_time <= stated_RTO,
    ]
    return all(checks)
# Run on a schedule (e.g. quarterly), not just
# once -- schema drift is ongoing, so verification
# has to be too.

What you see: The backup dashboard has been green for a year and a half; the moment an actual restore is needed, it succeeds mechanically but is missing data nobody knew was being silently dropped — the gap between "job succeeded" and "system is recoverable" was invisible for the entire period because nothing ever exercised the second half of that claim.

Why: A job's exit status can only tell you whether the process it was told to run completed — it has no way to know if the list of things it was told to back up is still complete, or whether the resulting file can rebuild a working system. Only an actual restore, checked against the live system, tests the claim a backup is actually being kept for.

Assumed recoverability vs. drilled recoverability

Backup job reports success

+File was written to storage
+Dashboard shows green
+Nobody has opened the file in a year

Backup was actually restored in a drill

−Restored into a real environment
−Schema/data verified against live system
−Restore time measured against RTO

What a green backup dashboard does and does not prove

What a green backup dashboard does and does not prove
Signal	What it actually proves	What it does NOT prove
Backup job exit code 0	The backup process ran to completion	The backup is complete, current, or restorable
Backup file size looks normal	Roughly the expected amount of data was written	The data inside is structurally valid or matches the live schema
Successful restore in a drill	This backup, restored today, produces a working system	The NEXT backup will restore just as cleanly after the next schema change
Standby promoted successfully in a drill	The failover path worked under test conditions	It will work identically during a real, unplanned incident with production load

Remember: A backup job reporting success only proves bytes were written — restoring it, and checking the result against the live system and the stated RTO, is the only thing that proves recoverability. The same applies to failover: a standby that has never actually taken traffic is untested by definition. Schedule restore/failover drills on a recurring cadence (and after schema/infra changes), because a single past success expires the moment the system underneath it changes.