Disaster Recovery

Disaster recovery is not one technique — it is a small toolkit of mechanisms, each protecting against a different failure blast radius, and a real DR strategy usually combines several. Backups are a copy of data taken at a point in time and stored somewhere independent of the live system, so a bad deploy, a bug that corrupts rows, or outright deletion can be undone by restoring from before it happened. Point-in-time recovery (PITR) is a refinement of backups: instead of only being able to restore to the moment of the last full snapshot, a continuous log of changes (write-ahead log, binlog, oplog) lets you restore to any specific second — critical when the disaster is "we discovered the corruption six hours after it happened" rather than "the whole server died." Replication (covered in depth in §22 — primary/replica, sync vs. async) keeps a second live copy up to date in near-real-time, protecting against hardware failure but not against corruption or deletion that replicates just as faithfully as good data. Multi-zone deployment spreads instances across a cloud provider's availability zones (independent power, cooling, and networking within one region) so a single data-center-level failure does not take the whole system down. Multi-region strategies go further, keeping a fully separate copy of the system in a geographically distant region, protecting against a failure that takes out an entire region — the rarest and most expensive failure to defend against, reserved for the highest-stakes systems.

// A layered DR posture, cheapest ring first
1. Backups (daily full + continuous change log for PITR)
2. Replication (§22): sync within a region, async cross-region
3. Multi-zone: instances + replicas spread across 3 AZs
4. Multi-region: standby copy in a second region, promoted on
   a declared regional disaster

RPO (Recovery Point Objective) and RTO (Recovery Time Objective) are the two numbers that turn "we have disaster recovery" into an actual, testable guarantee — without them, "disaster recovery" is just a feeling. RPO answers "how much data can we afford to lose?", expressed as a time window: an RPO of 1 hour means that after recovering from a disaster, the restored data can be missing at most the last hour's worth of writes. It is driven entirely by how the toolkit's backup/replication cadence works — continuous synchronous replication can approach an RPO of zero, nightly backups imply an RPO of up to 24 hours. RTO (Recovery Time Objective) answers a different question — "how long can the system be down?" — measured from the moment the disaster is declared to the moment the system is back up and serving traffic again. A system can have an excellent RPO (almost no data lost) and a terrible RTO (it takes six hours to actually bring the restored system back online), or vice versa — they are independent numbers, driven by different mechanisms, and a DR strategy is only complete when both are stated and both are actually achievable by the toolkit in place.

// Stating a DR requirement means stating both numbers
service: payments-ledger
RPO: 0        // synchronous replication, zero data loss tolerated
RTO: 5 min    // hot standby, automated failover already tested

service: internal-wiki
RPO: 24h      // nightly backup is acceptable
RTO: 4h       // manual restore from backup is acceptable

A backup job that reports "success" every night for two years tells you a file was written somewhere — it tells you nothing about whether that file can actually rebuild a working system. Backups fail silently in ways a completion status never surfaces: the backup format drifts out of sync with a schema migration, credentials to the restore target quietly expire, a backup completes but is missing a table that was added after the backup script was last touched, or the backup is technically valid but restoring it takes eleven hours against a four-hour RTO. None of these show up as a red X in a dashboard — the job exits 0 every time, right up until the moment someone actually needs the data back and discovers, mid-incident, that the thing they were relying on does not work. The same logic applies to failover: a standby database or a secondary region that has never actually taken production traffic is a hypothesis, not a capability, because the failover path itself (DNS cutover, connection string changes, cache warm-up, dependent services reconnecting) has its own failure modes that only show up when it is actually exercised.

// A restore/failover drill, scheduled like any other test
quarterly_dr_drill:
  1. Pick last night's backup (not a golden known-good one)
  2. Restore it into an isolated environment
  3. Run schema/row-count/smoke-test checks against it
  4. Measure restore time against the stated RTO
  5. Promote a standby and cut traffic over in staging
  6. Record pass/fail + timing; file a ticket for any gap