beginnerintermediateadvanced

AWS · Section 58

AWS Disaster and Failure Testing

Level: advanced
Read: 35 min
Concepts: 3

Every resilience mechanism in a design — a health check, a retry, a replica, a failover — is a claim that has not been tested, and the gap between designing for a fault and verifying it is where outages live. AWS names that gap as an anti-pattern directly. The discipline that closes it has a fixed shape: measure a steady state on what users actually experience, write the hypothesis as "if this fault occurs, this mitigation keeps this metric inside this bound", inject the fault, compare, and either fix the design or automate the experiment as a regression test in the pipeline. The faults worth injecting are a short list that barely varies between workloads — instance loss, an impaired Availability Zone, container replacement, a worker killed mid-job, a queue backlog, a cache node failure, a database failover, a third-party timeout, a broken DNS record, an expired credential, a deployment rollback — and past incidents are the best guide to which of them to drill first. The one that most often disproves its hypothesis is database failover, because the managed failover works and the application's connection pool, still holding connections to a writer that no longer answers, is what turns thirty seconds into ten minutes. Because fault injection is the only testing that can itself cause an outage, the safety work is the design: AWS FIS models an experiment as actions, targets and stop conditions, where the target selection is the blast radius, up to five CloudWatch alarms can stop the experiment, and a post action can return targets to their prior state.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

An untested mitigation is a belief; chaos engineering turns it into a measured result.
Steady state is measured on user-visible output, never on internal metrics.
A short, near-universal list of faults covers most real failure modes.
Blast radius is three settings decided before the experiment: targets, duration, stop conditions.
Experiments that pass become scheduled regressions, because every deploy can remove a mitigation.

What you will be able to do

Write a chaos experiment with a measurable steady state and a falsifiable hypothesis
Prioritise failure drills by frequency and damage using past incident data
Run a database failover game day and interpret the result honestly
Bound an FIS experiment so it cannot become an outage
Turn a passing experiment into an automated regression test

From an untested assumption to an automated regression

A mitigation in the design

health checks, retries, replicas, failover

Make it an experiment

steady state, hypothesis, verify

Pick the drill

ordered by frequency × damage

Inject it safely

targets, duration, stop conditions

Automate what passed

a scheduled test, not a one-off event

AWS Disaster and Failure Testing

Why happy-path testing cannot find resilience gaps, the catalogue of faults worth drilling, and how to inject them with a blast radius that is decided in advance.

Testing More Than the Happy Path

coreadvanced

Most testing proves the system works when everything else is working. That is the least interesting case, because it is the one that happens on its own. The useful question is what happens when a dependency is slow, a node disappears, or a credential expires — and the only way to know is to make those things happen on purpose while you are watching.

Think of it as

You designed mitigations — health checks, retries, timeouts, replicas, failover. Each one is a claim: "if X happens, the system does Y." An untested claim is a belief. Chaos engineering is the discipline of turning each belief into an experiment with a stated hypothesis and an observed result.

What we're doing: Turn one design assumption into an experiment with a result you can act on.

first-experiment.txttext

The assumption, taken from the design document:
  "If a task dies, ECS replaces it and users do not notice."

Nobody has ever tested this. It is a belief.

STEADY STATE — measured for a week first, not guessed
  1,200 requests/min, 99.95% success, p95 210 ms.
  Any experiment result is meaningless without this baseline.

HYPOTHESIS
  If one of four ECS tasks is stopped, the ALB health check
  removes it within 30 seconds, the service starts a replacement,
  and the 5xx rate stays under 0.05% for the whole window.

RUN — in pre-production first, at low traffic, one task only
  Stop one task. Watch the four numbers. Alarm armed to stop the
  experiment if 5xx crosses 1%.

RESULT — the hypothesis was wrong, usefully
  5xx spiked to 4% for 45 seconds. Cause: the ALB deregistration
  delay was 300 seconds, but the container had no graceful
  shutdown handler, so in-flight requests were killed rather than
  drained. The health check was fine; the shutdown path was not.

That gap was in production for eight months and no amount of
happy-path testing would have found it.

1: Design documents are full of sentences like this. Each one is a candidate experiment.
6: Without a baseline you cannot tell a bad result from a normal Tuesday — this is why steady state comes first.
13: The hypothesis names the mitigation, so a failure tells you which mitigation was wrong.
19: The valuable outcome is a disproved hypothesis: it converts an unknown into a specific, fixable defect.

Why this works: The gap between "we designed for this" and "we verified this" is where outages live, and it is invisible from inside normal testing because normal testing never creates the condition. One experiment per design assumption, run in the order of how likely and how damaging each fault is, converts that gap into a finite list of things to fix.

Running an experiment against a workload you already know is broken

Wrong

text

# "The retry logic is not implemented yet, but let us inject
# a dependency failure and see what happens."

Better

text

# Implement the retry logic and its bound first.
# Then experiment, to find out what you did NOT predict.

What you see: The experiment causes a real incident, teaches nothing anyone did not already know, and makes it politically much harder to run the next one.

Why: AWS states this directly: an experiment should be fail-safe and tolerated by the workload, and if you know the workload will fail, do not run it. Chaos engineering is for finding known-unknowns and unknown-unknowns; experimenting against a known break spends trust and blast radius to reconfirm something already on the backlog.

The chaos experiment loop

The loop only produces value if the steady state is measured before the fault and the result feeds back into the design. Skipping either end turns it into breaking things for their own sake.

Two kinds of testing, both needed

Two kinds of testing, both needed
Aspect	Resilience testing	Chaos engineering
Question it answers	Does the known behaviour work?	What happens that we did not predict?
Example	Fail over the database and confirm reconnection	Add 300 ms of latency to a dependency and watch the whole system
Expected result	Known — it is a regression test	Unknown — that is the point
Where it runs	Pre-production, in the pipeline	As close to production as possible, ideally production
Failure means	A regression was introduced	A gap in the design was found

Together

text

# AWS's hypothesis template, filled in
# "If {fault} occurs, the {workload} will {mitigation}
#  to maintain {metric}."

If the primary RDS instance fails, the orders workload will fail
over to the standby and reconnect, maintaining less than one
minute of database read or write errors.

Remember: Every mitigation in the design is an untested claim until you make the fault happen. Measure a steady state first, write the hypothesis as "if this fault, then this mitigation keeps this metric inside this bound", run it small, and measure what the user sees rather than what the servers report.

The Failure Drill Catalogue

coreadvanced

There is a short, finite list of things that actually go wrong in an AWS workload, and it is nearly the same list for every workload. Working through it as drills — one fault at a time, each with a hypothesis and a measured result — turns an open-ended fear into a checklist with a completion state.

Think of it as

Order the list by two numbers: how often the fault happens, and how much it hurts when it does. AWS says the same thing about prioritising experiments — start with a coarse high, medium, low categorisation using past incidents as the data. The top of that list is where the first drill goes, not the fault that is easiest to inject.

What we're doing: Run a game day for the drill most workloads fail: database failover.

failover-game-day.txttext

BEFORE
  Tell the operations, support and on-call teams the window and
  what to expect, and give them a way to call it off. An
  unannounced drill teaches you about your paging, not your system.

  Baseline for 30 minutes. Arm a stop condition on 5xx > 1%.

DURING — one action, watched closely
  Force the failover. Then watch four things, in this order:
    1. Does the endpoint resolve to the new writer, and how fast?
    2. Do application connections reconnect, or hang?
    3. Do in-flight transactions fail cleanly or corrupt state?
    4. Does the error rate return to baseline without intervention?

WHAT USUALLY BREAKS AT STEP 2
  The pool holds connections to the old writer. They are not
  closed by the failover — they simply stop answering. Without a
  connection health check or a bounded connection lifetime, every
  request waits for a TCP timeout, and the outage lasts far longer
  than the failover itself did.

AFTER
  Record the real numbers, not the plan: measured RTO, how long
  errors lasted, what needed a human. Compare against the stated
  RTO. If they disagree, one of the two is wrong and it is
  usually the document.

1: Announcing the drill is not cheating — the point is to test the system, and a surprise mostly tests the people.
7: One fault at a time. Two simultaneous faults produce a result you cannot attribute to either.
14: This is the specific failure that makes database failover the most valuable first drill: the database recovers and the application does not.
21: A measured RTO that contradicts the documented one is the most useful artifact a game day produces.

Why this works: Managed failover works reliably; what fails is the application's assumption that a connection which was open a second ago is still usable. That assumption is invisible in every other kind of testing, because nothing else takes a live connection away mid-flight. One drill finds it, and the fix — a bounded connection lifetime and a health check on checkout — is small.

Running drills only in an environment that does not look like production

Wrong

text

# Failover drill in staging: one AZ, one task, no traffic,
# a 2 GB database and no cache.

Better

text

# Pre-production sized like production, under synthetic load,
# then production during off-peak once the drill is understood.

What you see: Every drill passes in staging and the same fault causes an incident in production, because the behaviour that breaks only appears under real concurrency and real data volume.

Why: Most failure behaviour is load-dependent: connection pools only exhaust when they are in use, stampedes only happen with enough concurrent misses, and scaling policies only misbehave when there is something to scale. AWS recommends running experiments in or as close to production as possible for exactly this reason, starting in pre-production and moving toward production as confidence grows.

Which drill to run first

SevereContained

RareFrequent

Single instance or task dies: Happens weekly. Should be invisible — verify that it is.
Deployment rollback: Frequent and high-stakes: practise it before you need it at 3am.
Database failover: Uncommon, very visible. The classic first serious drill.
Queue backlog: Common under load. Tests whether workers scale on the right signal.
Third-party API timeout: Constant in practice. Tests timeouts, retries and fallbacks.
Cache node failure: Tests whether the cache is really optional.
Expired or revoked credentials: Rare, total, and usually undetected until it happens.
Availability Zone impairment: Rare and severe — the reason multi-AZ exists at all.
DNS misconfiguration: Rare, and takes everything down at once when it lands.

Eleven drills, what each one proves, and how it fails in practice

Eleven drills, what each one proves, and how it fails in practice
Drill	The claim it tests	How it usually fails
Terminate one instance	Auto Scaling replaces it and users do not notice	No graceful shutdown, so in-flight requests are killed rather than drained
Impair an Availability Zone	The other zones absorb the traffic	A single-AZ resource nobody noticed — a NAT gateway, a cache node, one subnet
Stop a container task	The service replaces it within the deregistration delay	Health check interval and deregistration delay were never tuned together
Kill a worker mid-job	The message returns to the queue and is reprocessed	The handler is not idempotent, so the retry duplicates the effect
Flood a queue	Workers scale out and drain the backlog	Scaling is on queue depth rather than backlog per worker, so it never converges
Fail a cache node	Requests fall through to the database	The database cannot take the full uncached load — a stampede, not a fallback
Force a database failover	The application reconnects within the RTO	A connection pool holds dead connections; nothing reconnects until a restart
Time out a third-party API	The call is bounded and degrades gracefully	No timeout set, so a slow provider consumes every request thread
Break a DNS record	Monitoring detects it before customers do	TTLs are long, so the rollback takes as long as the outage
Expire a credential	The failure is loud and specific	AccessDenied appears as a generic 500 with no attribution
Roll back a deployment	The previous version returns quickly	A forward-only migration has already made the old version unable to start

Together

text

# One drill written out, ready to run
DRILL      cache node failure
STEADY     1,200 rpm, 99.95% success, p95 210 ms, cache hit 94%
HYPOTHESIS Removing the primary cache node raises p95 to under
           600 ms and keeps success above 99.5%, because misses
           fall through to RDS and the connection pool absorbs it.
GUARDRAIL  Stop if 5xx > 1% or p95 > 2 s.
RESULT     __________
ACTION     __________

Remember: Eleven faults cover nearly everything: instance, AZ, container, worker, queue backlog, cache, database failover, third-party timeout, DNS, credentials, rollback. Order them by frequency and damage using past incidents, run one at a time against a measured baseline, and turn every drill that passes into a scheduled regression.

Fault Injection with a Controlled Blast Radius

coreadvanced

Fault injection means deliberately breaking part of a running system. AWS Fault Injection Service does this as a managed experiment: you declare the actions, the exact resources they apply to, and the CloudWatch alarms that stop the experiment if it goes further than intended. The guardrails are the difference between an experiment and an incident.

Think of it as

Three settings decide the blast radius before anything runs: which resources are targeted, for how long, and what stops it early. Decide all three in writing first. "We will see how it goes" is not a blast radius — it is an outage with a scheduled start time.

What we're doing: Write a first production experiment that cannot become an incident.

bounded-experiment.txttext

SCOPE — decided before anything is configured
  One ECS task out of twelve, in one service, in one account.
  Five minutes. Tuesday 14:00, which is a measured traffic trough.
  Two stop conditions, both on what a user would feel.

TARGETS — narrowed by tag, not by service
  Only tasks tagged Chaos=eligible are selectable, and exactly
  one is chosen. A resource created later is not automatically in
  scope, because it will not carry that tag by default.

STOP CONDITIONS — armed and verified first
  alarm: 5xx rate > 1% for 1 minute
  alarm: p99 latency > 2s for 1 minute
  Both were tested in pre-production by deliberately breaching
  them, so it is known that they fire and that FIS stops.

ROLLBACK
  ECS replaces the task automatically. The post action is a
  no-op here, which is itself a decision worth writing down —
  the recovery is the workload's own mechanism, and verifying
  that is the whole point of the experiment.

COMMUNICATION
  On-call, support and the service team know the window, know
  what they should not page for, and know who can call it off.

1: Every number here is chosen before the template is written. That ordering is what keeps the experiment small.
8: Tag-based selection means the blast radius does not silently grow as the fleet does.
15: An untested stop condition is not a guardrail. Breaching it on purpose in pre-production is the only way to know it works.
21: Naming the recovery mechanism explicitly is what turns the experiment into a test of a specific claim.

Why this works: Fault injection is the only kind of testing that can itself cause an outage, so the safety work is not overhead around the experiment — it is the experiment's design. The three template fields map directly onto the three questions worth answering in advance: what breaks, how much of it, and what makes us stop.

Targeting a whole service or Auto Scaling group by default

Wrong

text

targets:
  ecsTasks:
    resourceTags: { Service: orders }
    selectionMode: ALL

Better

text

targets:
  ecsTasks:
    resourceTags: { Service: orders, Chaos: eligible }
    selectionMode: COUNT(1)

What you see: What was meant to test one task's replacement takes the whole service down at once, which is a real outage rather than a result.

Why: AWS FIS carries out real actions on real resources — the selection mode is the difference between removing one member of a redundant set and removing the redundancy itself. `ALL` also grows silently: every task added later is in scope without anyone re-reading the template.

An experiment template is three decisions and one kill switch

Actions say what breaks, targets say how much, stop conditions say when to abandon the experiment. All three are declared before the experiment starts.

How to shrink the blast radius, in the order to apply them

How to shrink the blast radius, in the order to apply them
Control	What it bounds	A concrete setting
Environment	Who is affected at all	Pre-production first; production only once the drill is understood
Target selection	How many resources	One task, or 5% of instances by tag — not "all instances in the ASG"
Timing	How long, and when	A 5-minute action during off-peak, not an open-ended one at the daily peak
Stop conditions	How far it is allowed to go	Alarm on user-visible 5xx and p99 latency, not on internal metrics
Traffic	Whose requests are exposed	Synthetic traffic against a canary deployment when customer impact is too risky
Rollback	What is left behind	A post action returning targets to their prior state, plus a documented manual path
Communication	Who is surprised	Operations, support and on-call know the window and can call it off

Together

text

# The same experiment, twice — the difference is entirely in
# the targets and the stop conditions

# Blast radius: unbounded
targets: all EC2 instances where Environment=prod
duration: until stopped
stopConditions: none

# Blast radius: bounded
targets: EC2 instances where Environment=prod AND Chaos=eligible,
         selectionMode: COUNT(1)
duration: PT5M
stopConditions: alarm prod-5xx-rate > 1%
                alarm prod-p99-latency > 2s

Remember: Declare actions, targets and stop conditions before running anything. Bound the radius by tag and count rather than by service, keep the duration short and off-peak, arm CloudWatch alarms on user-visible signals and test that they fire, and run in pre-production before production.