beginnerintermediateadvanced

AWS · HA and Reliability

High Availability and Reliability

Redundancy and replacement, dependency isolation, quotas as ceilings, and the reliability pillar

The least redundant thing sets availabilitynot the most redundant one
Slow is worse than downa hang consumes the caller for as long as it lasts
Quotas bind at your target trafficraise them before, not during
Untested failover is a hypothesisAWS puts testing inside its definition

1.Redundancy, Health Checks, and Automated Replacement

High availability is four habits, not one setting. Run in more than one Availability Zone, check health continuously, replace unhealthy things automatically instead of repairing them, and remove every component whose failure takes the system with it. The fourth one is the hard part, because single points of failure are usually shared services nobody lists.

Key features

Redundancy only helps if failure is detected: an unhealthy instance still receiving traffic is worse than one that was never there
The health check must reflect what a user needs but must not depend on a shared downstream, or one dependency failure removes the whole fleet at once
Automated replacement beats repair — an Auto Scaling group replacing an instance is faster and more repeatable than any human diagnosis
Enumerate single points of failure by asking, for every component in the path, which AZ it is in and what happens to the others when that AZ fails
Untested failure handling is a hypothesis: the failover that has never been exercised is the one that finds its bug during the incident
AWS defines reliability as a workload performing its intended function correctly and consistently when expected to, including the ability to operate and test it through its whole lifecycle

Single points of failure people miss

Single points of failure people miss
Component	Why it hides	The fix
One NAT gateway	Traffic works normally until its AZ fails	One NAT per AZ, each private subnet routing to its own
A single cache node	Cache is "just performance" until sessions live there	Multi-AZ replication group, and no correctness dependency on the cache
One writable database	Multi-AZ RDS still has a single writer	Accept it, but rehearse failover and design for the reconnect
A shared internal service	Owned by another team, not on your diagram	Timeouts and a degraded mode when it is unavailable
One deployment pipeline	Not in the request path — until you cannot deploy a fix	A documented manual path for emergencies
A single Region	Multi-AZ is not multi-Region	A deliberate DR posture, not an assumption

The four habits, in the order they fail without each other

Redundancy

more than one of everything in the path

Health checks

so failure is noticed

Automated replacement

so it is fixed without a human

Tested failure modes

so you know it actually works

Signature

redundancy + health checks (no shared dependency) + automated replacement + tested failure modes

Redundancy, health checks that do not depend on shared downstreams, automated replacement rather than repair, and failure modes you have actually exercised. Then walk every component in the path and name its AZ — the one that appears once is your real availability.

2.Timeouts, Bounded Retries, Circuit Breakers, and Degradation

A distributed system fails when one slow dependency consumes all of a caller's capacity. The countermeasures are small and specific: cap how long you wait, cap how many times you retry, spread retries out so they do not synchronize, stop calling something that is clearly down, and serve a reduced answer rather than no answer.

Key features

A call with no timeout inherits the callee's worst case: threads or connections pile up until the caller is exhausted
Retries multiply load exactly when a dependency is struggling — three retries across three tiers is 27 requests for one user action
Jitter is what prevents synchronized retries: without it every caller retries at the same moment and recreates the spike
A circuit breaker fails fast after a threshold of failures, giving the dependency room to recover and freeing the caller's capacity immediately
Graceful degradation means a reduced answer beats an error — stale cached data, a hidden recommendations panel, a queued write acknowledged as pending
Backpressure is refusing or queueing work you cannot do yet — how a system sheds load instead of collapsing — and retries only make sense on idempotent operations, so idempotency is the prerequisite for every mechanism here

The mechanisms, and the failure each one stops

The mechanisms, and the failure each one stops
Mechanism	Stops	Typical setting
Timeout	Caller exhaustion from a slow callee	A little above the callee's p99, not its average
Bounded retries	Amplifying load during a failure	2 attempts total for a user-facing call
Exponential backoff + jitter	Synchronized retry storms	Base 100 ms, full jitter
Circuit breaker	Wasting capacity on a dependency that is down	Open after ~50% failures over a window
Bulkhead / separate pool	One dependency starving the others	A dedicated connection pool per downstream
Graceful degradation	A non-critical dependency failing the request	Serve stale, or omit the section
Backpressure / queue	Accepting more work than you can do	Bounded queue, reject or shed beyond it

Circuit breaker states

failure thresholdcrossedcool-downelapsedtrialsucceedstrial fails

Closed — calls pass through

start

Open — fail fast, no calls

Half-open — one trial call

Signature

timeout · bounded retries + jitter · circuit breaker · bulkhead pools · degrade · backpressure · idempotency

Timeout every call. Bound retries and add jitter. Break the circuit on a dependency that is down. Give each downstream its own pool. Degrade to a reduced answer instead of an error. And note that all of it presumes idempotency — retries are only safe on operations that tolerate duplicates.

3.Service Quotas as Reliability Constraints

Every AWS service has quotas — some adjustable, some fixed. A quota you have never thought about becomes an outage the moment traffic grows into it, and it fails in a way that looks nothing like a capacity problem: throttling errors, tasks stuck provisioning, or a scaling event that silently does not happen.

Key features

Quotas are hard (cannot be changed) or soft (increases can be requested), and the distinction matters most when you discover one during an incident
Quotas usually apply per account and per Region, so a workload spanning Regions has separate ceilings in each
AWS documents the mismatch pattern explicitly for Lambda: API Gateway defaults to 10,000 requests per second while Lambda defaults to 1,000 concurrency, so the front door admits more than the compute can serve
Quota increases are not instant — a request during an incident is not a mitigation
The Service Quotas console shows current utilization against each quota, which is what makes an alarm on approaching a limit possible
AWS recommends load testing precisely to identify which quotas will bind before the traffic that binds them arrives

Quotas that commonly become incidents

Quotas that commonly become incidents
Quota	Symptom when hit	Mitigation
Lambda concurrent executions	Throttling (429), asynchronous events retried then dead-lettered	Request an increase; use reserved concurrency to allocate deliberately
ENIs per VPC	Fargate tasks stuck in PROVISIONING	Fewer VPC-attached functions, or request an increase
Elastic IPs per Region	A NAT gateway or load balancer cannot be created	Request an increase before the Region build-out
EC2 vCPU limits per instance family	Auto Scaling cannot launch — capacity is available, permission is not	Raise the quota for the families you actually use
API rate limits on the control plane	Deployments and describe calls throttled	Back off and cache; do not poll describe APIs in a loop
Subnet free addresses	Tasks and ENIs cannot be placed	Size subnets generously; add secondary CIDRs

Command

text

# Quota work, as a checklist
1. List the quotas each service in the path has
2. Compare each against projected peak, not current average
3. Request increases for the ones that bind — before you need them
4. Alarm on utilization approaching the ceiling

Quotas are per account and per Region, and some are unchangeable. Enumerate them for every service in the path, compare against projected peak rather than current average, request increases in advance, and alarm on utilization — a quota increase requested mid-incident is not a mitigation.

4.Well-Architected Reliability Principles

The reliability pillar of the Well-Architected Framework is AWS's written answer to "what does a reliable workload look like". It defines reliability as a workload performing its intended function correctly and consistently when expected to — including the ability to operate and test it through its whole lifecycle — and the practices behind that are the ones this section has been building toward.

Key features

AWS defines reliability as the ability of a workload to perform its intended function correctly and consistently when it is expected to, including operating and testing it through its total lifecycle
Loosely coupled dependencies: a queue or event between two services means one can be unavailable without failing the other
Throttling is a reliability feature, not a limitation — refusing excess work protects the work already accepted
Fail fast beats waiting: an immediate error frees the caller's capacity, while a slow failure consumes it
Stateless services can be replaced, scaled, and load-balanced freely, which is what makes automated replacement possible at all
Trade-offs between pillars are expected and business-driven — AWS notes that security and operational excellence are generally not traded off against the others

Each principle, as a question to ask about a design

Each principle, as a question to ask about a design
Principle	The question	A failing answer sounds like
Loosely coupled dependencies	Which of our dependencies can be down without us being down?	"All of them are required"
Throttling	What do we do with more work than we can handle?	"We accept it and hope"
Bounded retries	How many times, and with what backoff?	"Until it succeeds"
Fail fast	How long do we wait before giving up?	"There is no timeout"
Timeouts	Is every outbound call bounded?	"The defaults are probably fine"
Stateless services	What breaks if this instance disappears right now?	"The sessions on it"

Command

text

# The six pillars, for context (§51 covers them in full)
operational excellence · security · reliability
performance efficiency · cost optimization · sustainability

Reliability is a workload doing its job correctly and consistently, including being testable through its lifecycle. Loosen coupling, throttle deliberately, bound retries, fail fast, timeout everything, and keep services stateless — then review a design against those six as questions, not as advice.

Spotted a mistake on this page? Tell us.

Read Sheet