beginnerintermediateadvanced

AWS · Section 47

Disaster Recovery

Level: advanced
Read: 35 min
Concepts: 4

AWS groups disaster recovery into four approaches of rising cost and falling recovery time: backup and restore, which rebuilds infrastructure and code in the recovery Region and therefore depends on infrastructure as code; pilot light, which keeps data replicated and core stores always on while application servers stay switched off; warm standby, a scaled-down but fully functional copy that can serve traffic immediately; and multi-site active/active, where there is no failover step because the workload already runs everywhere. Which one you need comes from two numbers — RPO, how much data you can lose, bounded by backup or replication frequency, and RTO, how long you can be down, bounded by the slowest recovery step including detection and decision — and both are per-workload business decisions rather than a company-wide setting. Whatever the strategy, the plan is only real once rehearsed: test restores rather than backup creation, start the clock at the failure, and expect DR-Region quotas and departed runbook owners among the first findings. AWS recommends using only data plane operations during failover, since control planes have lower availability design goals, and warns against fully automatic failover because recovery time and recovery point are always above zero. Finally, multi-AZ and multi-Region answer different questions: multi-AZ is synchronous, largely automatic and the production baseline, while multi-Region is asynchronous, deliberate, and expensive — with cross-Region backups usually the right step between them.

This section

Disaster Recovery of Workloads on AWS | AWS EventsAWS Events

What is true here

Backup/restore → pilot light → warm standby → active/active, rising in cost and falling in recovery time.
Pilot light needs action before it can serve; warm standby serves immediately at reduced capacity.
RPO is set by backup/replication frequency; RTO by the slowest recovery step, including detection and decision.
Test restores and rehearse failovers — quotas and organisational gaps are the usual findings.
Multi-AZ is the synchronous, automatic baseline; multi-Region is an asynchronous, deliberate, per-workload decision.

What you will be able to do

Choose a DR strategy from an RTO and RPO rather than the other way round
Explain the difference between pilot light and warm standby precisely
Distinguish replication (Region loss) from backup (corruption and deletion)
Design a rehearsal that measures true end-to-end RTO, and automate restore testing
Say what multi-AZ protects against and where multi-Region becomes necessary

From objectives to a recovery you have actually performed

RPO and RTO

The four strategies

Multi-AZ vs multi-Region

Rehearsed and measured

Disaster Recovery

The four strategies and their trade-offs, the two objectives that choose between them, testing recovery for real, and the multi-AZ versus multi-Region distinction.

The Four DR Strategies

coreadvanced

AWS groups disaster recovery into four approaches, from cheapest and slowest to most expensive and fastest. Backup and restore keeps copies and rebuilds. Pilot light keeps the data replicated and the core infrastructure provisioned but switched off. Warm standby keeps a scaled-down but running copy. Multi-site active/active runs everywhere at once, so there is no failover at all.

Think of it as

Think of a spare car. Backup and restore is having the parts in a crate. Pilot light is having the car assembled but not started. Warm standby is having it idling in the driveway. Active/active is already driving both cars. The cost rises at every step, and so does the speed you can leave.

What we're doing: Choose between pilot light and warm standby for a workload with a 30-minute RTO.

pilot-vs-warm.txttext

Pilot light: data is replicated and current. At failover you deploy the
application tier, wait for it to start, warm caches, and cut traffic over.
Realistic: 20-45 minutes, and the variance is in steps nobody rehearsed.

Warm standby: a scaled-down copy is already running and already passing
health checks. At failover you scale it up and cut traffic over.
Realistic: 5-15 minutes, and the path is exercised continuously because
the environment is always alive.

Decision: with a 30-minute RTO, pilot light might make it. Warm standby
makes it with margin, and it is the only one of the two you can prove.

1: The technical difference is small; the confidence difference is not. Pilot light's failover path is only exercised when you deliberately exercise it.
6: AWS states this distinction directly: pilot light needs action before it can process requests, warm standby can serve immediately at reduced capacity.
10: "Might make it" is not an RTO. The strategy you can rehearse cheaply is worth more than the one that is theoretically fast enough.

Why this works: The four strategies differ in what is already running, and therefore in how much of the recovery path is unproven at the moment you need it. Cost buys speed, and it also buys certainty — which is usually the more valuable half.

Depending on control plane operations during failover

Wrong

text

# Failover plan: create the Auto Scaling group, create the load
# balancer, update the Route 53 weighted records

Better

text

# Pre-create the resources. At failover, use only data plane
# operations — Route 53 health check toggles (ARC), not record edits.

What you see: The failover runbook stalls because a control plane API is degraded in exactly the event that triggered the failover.

Why: AWS states that data planes typically have higher availability design goals than control planes, and recommends using only data plane operations as part of failover. A plan that creates resources during a disaster depends on the part of AWS most likely to be affected by it.

Four strategies, cheapest first

Backup and restore

Data copied; infrastructure and code redeployed at failover. Hours.

Pilot light

Data replicated continuously, core infrastructure provisioned but off. Tens of minutes.

Warm standby

Scaled-down but fully functional and serving-capable. Minutes.

Multi-site active/active

Running everywhere; no failover step exists. Near zero.

The four strategies compared

The four strategies compared
Strategy	What is running in the DR Region	Cost and complexity
Backup and restore	Backups only	Lowest — storage plus a tested rebuild path
Pilot light	Data stores and replication; servers deployed but off	Low — storage, replication, and deployment discipline
Warm standby	A scaled-down but fully functional copy	Moderate — you pay for the running copy
Hot standby	Full capacity, no traffic	High — statically stable, no Auto Scaling dependency at failover
Multi-site active/active	Full capacity, serving traffic	Highest — plus the hardest data-consistency design

Together

text

# The write strategy is what makes active/active hard
write global    -> all writes to one Region (Aurora global database)
write local     -> writes anywhere (DynamoDB global tables, last-writer-wins)
write partitioned -> writes routed by partition key, to avoid conflicts

Remember: Backup and restore (rebuild), pilot light (data live, servers off), warm standby (scaled-down but serving), active/active (no failover at all). Cost buys speed and, more importantly, a path you have proven. Use only data plane operations at failover, and never mistake replication for backup.

RPO and RTO

coreintermediate

RPO — recovery point objective — is how much data you can afford to lose, measured in time. RTO — recovery time objective — is how long you can afford to be down. Both are business decisions expressed as numbers, and they are what pick the DR strategy: the strategy does not come first and produce them.

Think of it as

Draw a line at the moment of the disaster. RPO looks backwards: how far back does the last usable copy sit? RTO looks forwards: how long until service resumes? Backup frequency sets one, and the failover mechanism sets the other.

What we're doing: See why "we take daily backups" and "we need an RPO of five minutes" cannot both be true.

rpo-arithmetic.txttext

Stated requirement: RPO of 5 minutes.
Actual mechanism: an automated snapshot at 02:00 each day.

A failure at 16:30 loses 14 hours and 30 minutes of writes.

The requirement was never met and nothing reported that, because RPO is
not a setting anywhere — it is an emergent property of the backup
schedule that nobody compared against the stated number.

1: Requirements like this are usually written once and never reconciled with the mechanism that would have to deliver them.
5: This is the entire value of writing the objectives down: they become a claim you can check against the configuration.

Why this works: RPO and RTO are only useful if someone compares them against the actual mechanism. Written down and never checked, they create a false sense of coverage that is more dangerous than having no target at all.

Measuring RTO from "we started recovering" instead of from the failure

Wrong

text

# "Restore takes 40 minutes, so our RTO is 40 minutes."

Better

text

# RTO = detection + decision + restore + validation + cutover.
# Measure it end to end in a rehearsal.

What you see: The rehearsal shows a 40-minute restore, and the real incident takes three hours — most of it spent noticing, escalating, and deciding to declare a disaster.

Why: Users experience downtime from the moment service stops, not from the moment engineers start working. Detection and decision are frequently the largest components, and they are the ones a restore-only rehearsal never measures.

RPO and RTO around a disaster

02:00
Last usable recovery point
The most recent backup or replicated state
02:47
Disaster
Everything after the recovery point is lost — this gap is the RPO
02:52
Detected and declared
Detection time counts toward RTO
03:20
Service restored
Disaster to restored is the RTO

What each objective is actually bought with

What each objective is actually bought with
Objective	Set by	Buy a smaller number with
RPO ≈ 24 h	Daily snapshots	More frequent snapshots
RPO ≈ minutes	Frequent snapshots or log shipping	Point-in-time recovery
RPO ≈ seconds	Continuous cross-Region replication	Aurora global database, DynamoDB global tables, S3 replication
RTO ≈ hours	Rebuild from backups	Infrastructure as code, tested restores
RTO ≈ tens of minutes	Pilot light	Pre-provisioned core infrastructure
RTO ≈ minutes	Warm standby	An always-running scaled-down copy

Together

text

# Objectives per workload, not per company
payments ledger : RPO 0-5 s   RTO 5 min   -> active/active or hot standby
customer app    : RPO 5 min   RTO 30 min  -> warm standby
internal admin  : RPO 24 h    RTO 8 h     -> backup and restore
marketing site  : RPO 24 h    RTO 24 h    -> backup and restore

Remember: RPO = how much data you can lose (set by backup or replication frequency). RTO = how long you can be down (set by the slowest recovery step, including detection and decision). Set them per workload, then check the mechanism actually delivers them.

Designing and Testing Recovery Procedures

coreadvanced

A backup you have never restored is a guess. A failover you have never performed is a plan. The only way to know your RTO is to run the recovery end to end, on a schedule, and measure it — which also converts the runbook from a document into something people have actually done.

Think of it as

Recovery is a skill, not an artifact. The team that has restored a database four times this year will do it calmly at 3am; the team with an excellent document and no practice will spend the first hour discovering that step three no longer applies.

What we're doing: Find out what a rehearsal discovers that a document cannot.

what-the-drill-found.txttext

Quarterly DR drill, target RTO 60 minutes. Actual: 3 hours 40 minutes.

Finding 1: the runbook referenced an AMI that lifecycle policy had
deleted four months earlier. 50 minutes lost rebuilding it.

Finding 2: the DR account's EC2 vCPU quota was the default, far below
production capacity. 35 minutes lost requesting and waiting.

Finding 3: the runbook's owner had left the company; nobody knew who
could approve the DNS cutover. 45 minutes lost finding someone.

None of the three is a technology problem. All three would have happened
during a real disaster.

2: The gap between target and actual is the finding. A drill that hits its target teaches you less than one that misses it badly.
5: Quotas in the DR Region are the single most common drill finding — AWS explicitly warns to ensure DR-Region quotas are high enough to scale to production capacity.
8: Organisational dependencies are invisible in a technical review and dominate real recovery times.

Why this works: A rehearsal finds the things a document cannot contain: resources that have since been deleted, quotas that were never raised, and people who are no longer there. Each is trivial to fix once known, and none is discoverable by reading.

Testing that the backup exists rather than that it restores

Wrong

text

# Monitoring: alarm if no snapshot was created in the last 24 hours

Better

text

# Automated restore into an isolated account, integrity checks, and a
# metric for both success and duration

What you see: Snapshots have been taken faithfully for a year; the first restore attempt reveals a table that has been excluded from the backup selection since a migration.

Why: Backup creation and backup usability are different properties, and only the first one is monitored by default. AWS recommends periodic restores partly for exactly this and partly because restore is a control plane operation that may be unavailable in the disaster that requires it.

A recovery rehearsal that is worth running

Declare the scenario

Region loss, data corruption, or account compromise — they have different answers

Start the clock at the failure

Not at the moment engineers start working

Follow the runbook as written

Every improvisation is a bug in the runbook

Validate, do not just start

A service that boots but serves wrong data has not recovered

Record the real RTO and RPO

Compare against the target, and fix the gap or the target

Rehearsal levels, cheapest first

Rehearsal levels, cheapest first
Level	What it proves	Frequency
Restore a backup into a scratch account	The backup is readable and complete	Automated, continuous
Restore and run integrity checks	The data is usable, not just present	Weekly or monthly
Failover a database in staging	The application reconnects correctly	Each release cycle
Full DR drill in the recovery Region	The end-to-end RTO, including decisions	Quarterly or half-yearly
Production failover in a low window	The plan works on the real system	Annually, once the drills pass

Together

text

# Automated restore testing, as a scheduled job
# 1. Restore the latest snapshot into an isolated account
# 2. Run integrity checks (row counts, checksums, a real query)
# 3. Emit a CloudWatch metric: RestoreSucceeded 1|0, RestoreDurationSeconds
# 4. Alarm on the metric — a failing restore is an incident, quietly

Remember: Test the restore, not the backup. Start the clock at the failure, not at the response. Rehearse on a schedule and record the real RTO. Automate every failover step but keep the decision human — and expect DR-Region quotas and departed runbook owners to be your first two findings.

Multi-AZ vs Multi-Region Recovery

standardadvanced

Multi-AZ protects against losing one data centre and is mostly automatic, synchronous, and cheap. Multi-Region protects against losing a whole Region and is manual, asynchronous, and expensive. They are answers to different questions, and multi-AZ — which almost every production workload should already have — is not a weaker form of multi-Region.

Think of it as

Multi-AZ is redundancy inside one building complex: fast links, shared administration, and the failure it survives is one building. Multi-Region is a second site in another city: everything is further away, slower, and separately administered — which is exactly why it survives events the first arrangement cannot.

text

# The sequence almost every workload should follow
1. Multi-AZ everywhere in production          (baseline)
2. Cross-Region backups                        (cheap, protects the data)
3. Pilot light or warm standby, if RTO demands it
4. Active/active, only where the business case is explicit

Treating multi-Region as the next step after multi-AZ for every workload

Wrong

text

# "We are multi-AZ, so multi-Region is the natural upgrade."

Better

text

# Multi-AZ is the baseline. The next step is usually cross-Region
# backups, then a DR strategy sized to that workload's RTO and RPO.

What you see: A second Region is stood up, drifts within months because deployments only ever touch the primary, and would not work if it were ever needed — while the cost continues.

Why: A second Region only helps if it is kept current, which means every deployment, every configuration change, and every quota increase has to apply there too. Without that discipline it becomes an expensive, decaying copy — whereas cross-Region backups protect the data at a fraction of the cost and cannot silently rot in the same way.

What each posture actually gives you

What each posture actually gives you
Aspect	Multi-AZ	Multi-Region
Protects against	Loss of one Availability Zone	Loss of an entire Region
Replication	Synchronous (RDS Multi-AZ)	Asynchronous — RPO is greater than zero
Failover	Largely automatic	Deliberate, usually manually initiated
Latency between sites	Low, single-digit milliseconds	Tens to hundreds of milliseconds
Cost	Modest — often a checkbox	Substantial — duplicate everything
Typical posture	The production baseline	A business decision, per workload

Together

text

# What has to be duplicated for a second Region to be usable
VPC, subnets, route tables, security groups
AMIs and container images (copied, not referenced)
Secrets and parameters (replicated, or re-created)
Certificates (regional, except CloudFront's us-east-1 requirement)
Quotas raised to production levels
DNS records and a failover mechanism

Remember: Multi-AZ: synchronous, automatic, cheap, survives losing one data centre — the production baseline. Multi-Region: asynchronous, deliberate, expensive, survives losing a Region — a per-workload business decision. Cross-Region backups sit between them and are usually the right second step.