AWS Disaster and Failure Testing cheat sheet · AWS

1.Testing More Than the Happy Path

Most testing proves the system works when everything else is working. That is the least interesting case, because it is the one that happens on its own. The useful question is what happens when a dependency is slow, a node disappears, or a credential expires — and the only way to know is to make those things happen on purpose while you are watching.

Key features

AWS defines chaos engineering, quoting the Principles of Chaos, as experimenting on a system to build confidence in its capability to withstand turbulent conditions in production
AWS names a common anti-pattern directly: designing for resiliency but not verifying how the workload functions as a whole when faults occur
An experiment starts from a measurable steady state — AWS's example is 300 TPS at 99% success and 500 ms round trip — that you observe throughout
The hypothesis names the fault and the mitigation: if this fault occurs, the workload will do this, and this metric stays within this bound
AWS advises not running an experiment you already know will fail — chaos engineering is for finding unknowns, not confirming a known break
An experiment that passes should become an automated regression test in the CI/CD pipeline, not a one-off event

Two kinds of testing, both needed

Two kinds of testing, both needed
Aspect	Resilience testing	Chaos engineering
Question it answers	Does the known behaviour work?	What happens that we did not predict?
Example	Fail over the database and confirm reconnection	Add 300 ms of latency to a dependency and watch the whole system
Expected result	Known — it is a regression test	Unknown — that is the point
Where it runs	Pre-production, in the pipeline	As close to production as possible, ideally production
Failure means	A regression was introduced	A gap in the design was found

The chaos experiment loop

The loop only produces value if the steady state is measured before the fault and the result feeds back into the design. Skipping either end turns it into breaking things for their own sake.

Signature

steady state → hypothesis → experiment (scoped, with stop conditions) → verify → improve → automate

Every mitigation in the design is an untested claim until you make the fault happen. Measure a steady state first, write the hypothesis as "if this fault, then this mitigation keeps this metric inside this bound", run it small, and measure what the user sees rather than what the servers report.

2.The Failure Drill Catalogue

There is a short, finite list of things that actually go wrong in an AWS workload, and it is nearly the same list for every workload. Working through it as drills — one fault at a time, each with a hypothesis and a measured result — turns an open-ended fear into a checklist with a completion state.

Key features

AWS FIS provides preconfigured faults across EC2, ECS, EKS and RDS, including termination, forced failovers, CPU and memory stress, throttling, latency and packet loss
Past post-incident analyses are the best source of which faults are frequent and which are damaging — AWS names neglecting them as an anti-pattern
Application-level drills can start with simple stressors such as memory and CPU exhaustion before moving to network faults
Third-party dependency drills work by blocking access to the provider for a set duration, from seconds to hours
Degradation faults — added latency, dropped packets, DNS resolution failure — often expose more than outright termination does
A drill that passes becomes an automated regression test in the pipeline; a drill that fails becomes a design change

Eleven drills, what each one proves, and how it fails in practice

Eleven drills, what each one proves, and how it fails in practice
Drill	The claim it tests	How it usually fails
Terminate one instance	Auto Scaling replaces it and users do not notice	No graceful shutdown, so in-flight requests are killed rather than drained
Impair an Availability Zone	The other zones absorb the traffic	A single-AZ resource nobody noticed — a NAT gateway, a cache node, one subnet
Stop a container task	The service replaces it within the deregistration delay	Health check interval and deregistration delay were never tuned together
Kill a worker mid-job	The message returns to the queue and is reprocessed	The handler is not idempotent, so the retry duplicates the effect
Flood a queue	Workers scale out and drain the backlog	Scaling is on queue depth rather than backlog per worker, so it never converges
Fail a cache node	Requests fall through to the database	The database cannot take the full uncached load — a stampede, not a fallback
Force a database failover	The application reconnects within the RTO	A connection pool holds dead connections; nothing reconnects until a restart
Time out a third-party API	The call is bounded and degrades gracefully	No timeout set, so a slow provider consumes every request thread
Break a DNS record	Monitoring detects it before customers do	TTLs are long, so the rollback takes as long as the outage
Expire a credential	The failure is loud and specific	AccessDenied appears as a generic 500 with no attribution
Roll back a deployment	The previous version returns quickly	A forward-only migration has already made the old version unable to start

Which drill to run first

SevereContained

RareFrequent

Single instance or task dies: Happens weekly. Should be invisible — verify that it is.
Deployment rollback: Frequent and high-stakes: practise it before you need it at 3am.
Database failover: Uncommon, very visible. The classic first serious drill.
Queue backlog: Common under load. Tests whether workers scale on the right signal.
Third-party API timeout: Constant in practice. Tests timeouts, retries and fallbacks.
Cache node failure: Tests whether the cache is really optional.
Expired or revoked credentials: Rare, total, and usually undetected until it happens.
Availability Zone impairment: Rare and severe — the reason multi-AZ exists at all.
DNS misconfiguration: Rare, and takes everything down at once when it lands.

Signature

instance · AZ · container · worker · queue backlog · cache · DB failover · 3rd-party timeout · DNS · credentials · rollback

Eleven faults cover nearly everything: instance, AZ, container, worker, queue backlog, cache, database failover, third-party timeout, DNS, credentials, rollback. Order them by frequency and damage using past incidents, run one at a time against a measured baseline, and turn every drill that passes into a scheduled regression.

3.Fault Injection with a Controlled Blast Radius

Fault injection means deliberately breaking part of a running system. AWS Fault Injection Service does this as a managed experiment: you declare the actions, the exact resources they apply to, and the CloudWatch alarms that stop the experiment if it goes further than intended. The guardrails are the difference between an experiment and an incident.

Key features

AWS FIS carries out real actions on real resources, and AWS strongly recommends a planning phase and a pre-production run before experimenting in production
An experiment template holds three things: actions, targets, and stop conditions
A target can be specific resources or a group selected by criteria such as tags or state — which is how the blast radius is bounded
A stop condition is a CloudWatch alarm; if it triggers while the experiment runs, FIS stops the experiment
FIS supports up to five stop conditions per experiment template
A rollback configuration (a post action) returns the target to the state it was in before the action ran

How to shrink the blast radius, in the order to apply them

How to shrink the blast radius, in the order to apply them
Control	What it bounds	A concrete setting
Environment	Who is affected at all	Pre-production first; production only once the drill is understood
Target selection	How many resources	One task, or 5% of instances by tag — not "all instances in the ASG"
Timing	How long, and when	A 5-minute action during off-peak, not an open-ended one at the daily peak
Stop conditions	How far it is allowed to go	Alarm on user-visible 5xx and p99 latency, not on internal metrics
Traffic	Whose requests are exposed	Synthetic traffic against a canary deployment when customer impact is too risky
Rollback	What is left behind	A post action returning targets to their prior state, plus a documented manual path
Communication	Who is surprised	Operations, support and on-call know the window and can call it off

An experiment template is three decisions and one kill switch

Actions say what breaks, targets say how much, stop conditions say when to abandon the experiment. All three are declared before the experiment starts.

Signature

FIS experiment template = actions + targets + stop conditions (≤ 5 CloudWatch alarms)

Declare actions, targets and stop conditions before running anything. Bound the radius by tag and count rather than by service, keep the duration short and off-peak, arm CloudWatch alarms on user-visible signals and test that they fire, and run in pre-production before production.