beginnerintermediateadvanced

AWS · Section 46

High Availability and Reliability

Level: advanced
Read: 35 min
Concepts: 4

High availability is four habits: run in more than one Availability Zone, check health continuously, replace unhealthy components automatically rather than repairing them, and remove every single point of failure — which means walking the request path and naming the Availability Zone of each component, because availability is bounded by the least redundant one, not the most. The health check itself must not depend on a shared downstream, or one dependency failure removes the whole fleet at once. Beyond redundancy sits dependency isolation: a slow dependency is worse than a dead one because it consumes the caller's capacity for as long as it hangs, so every outbound call needs a timeout, retries need a limit and jitter, a circuit breaker frees capacity when a dependency is clearly down, separate pools stop one downstream starving the others, and degrading to a reduced answer beats returning an error. All of that presumes idempotency, since retries are only safe on operations that tolerate duplicates. Service quotas are the ceiling underneath everything — per account, per Region, some unchangeable — and they bind at exactly the traffic level a scaling design was built for. Finally, the Well-Architected reliability pillar states AWS's own definition and the principles behind it: loose coupling, deliberate throttling, bounded retries, fail-fast behaviour, timeouts everywhere, and stateless services.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

Availability is set by the least redundant component in the path — enumerate them by AZ.
Health checks must not depend on shared downstreams, or every replica fails together.
Timeouts, bounded retries with jitter, circuit breakers, bulkheads, and degradation each stop a specific failure.
Quotas are per account and per Region, and bind at the traffic level you were scaling toward.
AWS defines reliability to include testing the workload through its lifecycle — untested failover does not count.

What you will be able to do

Audit a design for single points of failure by naming every component's Availability Zone
Write a health check that reflects service health without coupling to a shared dependency
Apply timeouts, bounded retries with jitter, circuit breakers, and bulkheads to an outbound call
Design a degraded mode for an optional dependency instead of failing the request
Enumerate the quotas in a workload's path and raise the binding ones before they bind

From redundancy to a design you can review

Redundancy, health checks, replacement

Timeouts, retries, breakers, degradation

Quotas as ceilings

Well-Architected reliability principles

High Availability and Reliability

Redundancy and automated replacement, isolating dependencies so failures do not cascade, quotas as hard ceilings, and the reliability pillar as a review checklist.

Redundancy, Health Checks, and Automated Replacement

coreadvanced

High availability is four habits, not one setting. Run in more than one Availability Zone, check health continuously, replace unhealthy things automatically instead of repairing them, and remove every component whose failure takes the system with it. The fourth one is the hard part, because single points of failure are usually shared services nobody lists.

Think of it as

Availability is set by the weakest link in the path, not the strongest. Three AZs of compute behind one NAT gateway, one cache node, and one bastion is a single-AZ system wearing a multi-AZ diagram.

What we're doing: Test a failure mode instead of assuming it works.

test-the-failover.txttext

Assumption: "RDS Multi-AZ fails over automatically, so we are covered."

Test (in staging, then in production during a low window): trigger a
failover with reboot --force-failover and watch.

Observed: failover completes in about 60 seconds. The application does
not recover for 15 minutes, because the connection pool keeps handing
out sockets to the old endpoint until the JVM's DNS cache expires.

Fix: set the DNS TTL handling in the driver, and add a pool validation
query. Neither would have been found without the test.

1: The assumption is true about the database. It is silent about the application, which is where the outage actually happened.
5: Sixty seconds of database failover became fifteen minutes of user-visible downtime — an application-layer problem that only a real failover exposes.
9: Both fixes are small. Neither is discoverable from a diagram, a runbook, or a code review.

Why this works: The managed service usually does its part correctly; the gap is almost always in how the application reacts. That gap is only visible when the failover actually happens, which is why AWS puts testing the workload through its lifecycle inside its own definition of reliability.

A health check that queries the database

Wrong

text

# GET /health -> SELECT 1 against the primary; 500 if it fails

Better

text

# GET /healthz -> the process is up and can serve. Dependency status is
# a separate, monitored endpoint that does not gate load balancing.

What you see: A two-second database blip marks every task unhealthy simultaneously, the load balancer removes all of them, and a recoverable hiccup becomes a full outage that then has to cold-start.

Why: The health check decides whether an instance receives traffic. Coupling it to a shared dependency converts any dependency failure into a correlated, fleet-wide removal — the exact opposite of what redundancy is for, since all replicas fail the check at the same instant.

The four habits, in the order they fail without each other

Redundancy

more than one of everything in the path

Health checks

so failure is noticed

Automated replacement

so it is fixed without a human

Tested failure modes

so you know it actually works

Single points of failure people miss

Single points of failure people miss
Component	Why it hides	The fix
One NAT gateway	Traffic works normally until its AZ fails	One NAT per AZ, each private subnet routing to its own
A single cache node	Cache is "just performance" until sessions live there	Multi-AZ replication group, and no correctness dependency on the cache
One writable database	Multi-AZ RDS still has a single writer	Accept it, but rehearse failover and design for the reconnect
A shared internal service	Owned by another team, not on your diagram	Timeouts and a degraded mode when it is unavailable
One deployment pipeline	Not in the request path — until you cannot deploy a fix	A documented manual path for emergencies
A single Region	Multi-AZ is not multi-Region	A deliberate DR posture, not an assumption

Together

text

# The question that finds them all
For every component in the request path:
  1. How many of it are there?
  2. Which AZ is each one in?
  3. What happens to the other AZs when that one fails?

Remember: Redundancy, health checks that do not depend on shared downstreams, automated replacement rather than repair, and failure modes you have actually exercised. Then walk every component in the path and name its AZ — the one that appears once is your real availability.

Timeouts, Bounded Retries, Circuit Breakers, and Degradation

coreadvanced

A distributed system fails when one slow dependency consumes all of a caller's capacity. The countermeasures are small and specific: cap how long you wait, cap how many times you retry, spread retries out so they do not synchronize, stop calling something that is clearly down, and serve a reduced answer rather than no answer.

Think of it as

Every outbound call is a loan of your own capacity to someone else's reliability. A timeout is the repayment date, a retry limit is the credit limit, and a circuit breaker is refusing to lend to a borrower who has stopped paying.

What we're doing: Trace how one slow dependency takes down a service that does not depend on it for correctness.

cascade.txttext

The recommendations service starts responding in 30 seconds instead of
80 milliseconds. It is not required to render the page.

The product page calls it with no timeout. Each request now holds a
worker thread for 30 seconds instead of 80 ms.

At 200 workers and 50 requests per second, every worker is occupied
within four seconds. The product page stops serving — including for
users who would never have seen a recommendation.

With an 800 ms timeout and a fallback to "no recommendations", the page
renders in under a second, without that panel.

1: Slow is worse than down. A dependency that fails fast frees the caller immediately; one that hangs consumes capacity for as long as it hangs.
4: The missing timeout is the whole bug. Everything after this line is arithmetic.
8: The blast radius is the entire page, not the feature — which is why an optional dependency needs a timeout more than a required one does.

Why this works: Cascading failure is not caused by the dependency breaking; it is caused by the caller having no bound on how much of itself it will spend waiting. Timeout plus fallback converts a total outage into a missing panel.

Retrying without a limit or without jitter

Wrong

python

while True:
    try:
        return call_downstream()
    except Exception:
        time.sleep(1)      # every caller, every second, together

Better

python

for attempt in range(3):
    try:
        return call_downstream()
    except Exception:
        time.sleep(random.uniform(0, 0.1 * 2 ** attempt))
raise DownstreamUnavailable()

What you see: The dependency recovers briefly, is immediately hit by every caller's synchronized retry at the same instant, and falls over again — repeatedly, in a pattern that looks like it is flapping on its own.

Why: Unbounded retries turn a partial failure into sustained overload, and a fixed sleep synchronizes every caller onto the same schedule. Jitter spreads the retries across the interval so the recovering dependency sees a ramp instead of a wall.

Circuit breaker states

Closed — calls pass through

start

Open — fail fast, no calls

Half-open — one trial call

The mechanisms, and the failure each one stops

The mechanisms, and the failure each one stops
Mechanism	Stops	Typical setting
Timeout	Caller exhaustion from a slow callee	A little above the callee's p99, not its average
Bounded retries	Amplifying load during a failure	2 attempts total for a user-facing call
Exponential backoff + jitter	Synchronized retry storms	Base 100 ms, full jitter
Circuit breaker	Wasting capacity on a dependency that is down	Open after ~50% failures over a window
Bulkhead / separate pool	One dependency starving the others	A dedicated connection pool per downstream
Graceful degradation	A non-critical dependency failing the request	Serve stale, or omit the section
Backpressure / queue	Accepting more work than you can do	Bounded queue, reject or shed beyond it

Together

python

# Bounded, jittered, and only because the call is idempotent
for attempt in range(3):
    try:
        return client.get_prices(sku, timeout=0.8)   # not unbounded
    except Timeout:
        if attempt == 2:
            return cached_prices(sku)                # degrade, do not fail
        time.sleep(random.uniform(0, 0.1 * 2 ** attempt))   # full jitter

Remember: Timeout every call. Bound retries and add jitter. Break the circuit on a dependency that is down. Give each downstream its own pool. Degrade to a reduced answer instead of an error. And note that all of it presumes idempotency — retries are only safe on operations that tolerate duplicates.

Service Quotas as Reliability Constraints

standardadvanced

Every AWS service has quotas — some adjustable, some fixed. A quota you have never thought about becomes an outage the moment traffic grows into it, and it fails in a way that looks nothing like a capacity problem: throttling errors, tasks stuck provisioning, or a scaling event that silently does not happen.

Think of it as

A quota is a ceiling you cannot see until you hit your head on it. Scaling designs assume resources are elastic; quotas are the point where they stop being, and the failure arrives at exactly the traffic level you were scaling to handle.

text

# Quota work, as a checklist
1. List the quotas each service in the path has
2. Compare each against projected peak, not current average
3. Request increases for the ones that bind — before you need them
4. Alarm on utilization approaching the ceiling

Load testing the application but not the quotas

Wrong

text

# Load test hits 3,000 rps against a staging account with the same
# default quotas as production, and passes

Better

text

# Test at projected peak in an account with production quotas, and read
# the Service Quotas utilization view afterwards

What you see: Production throttles at a traffic level the load test cleared, because staging never generated enough concurrency to reach the same ceiling.

Why: A load test validates the application against the load, not the account against the quota — and the two ceilings are unrelated. AWS recommends load testing specifically to identify limiting quotas, which only works if the test reaches the levels that bind them.

Quotas that commonly become incidents

Quotas that commonly become incidents
Quota	Symptom when hit	Mitigation
Lambda concurrent executions	Throttling (429), asynchronous events retried then dead-lettered	Request an increase; use reserved concurrency to allocate deliberately
ENIs per VPC	Fargate tasks stuck in PROVISIONING	Fewer VPC-attached functions, or request an increase
Elastic IPs per Region	A NAT gateway or load balancer cannot be created	Request an increase before the Region build-out
EC2 vCPU limits per instance family	Auto Scaling cannot launch — capacity is available, permission is not	Raise the quota for the families you actually use
API rate limits on the control plane	Deployments and describe calls throttled	Back off and cache; do not poll describe APIs in a loop
Subnet free addresses	Tasks and ENIs cannot be placed	Size subnets generously; add secondary CIDRs

Together

text

# What is close to its ceiling right now?
aws service-quotas list-service-quotas --service-code lambda \
  --query "Quotas[].{name:QuotaName,value:Value,adjustable:Adjustable}"

Remember: Quotas are per account and per Region, and some are unchangeable. Enumerate them for every service in the path, compare against projected peak rather than current average, request increases in advance, and alarm on utilization — a quota increase requested mid-incident is not a mitigation.

Well-Architected Reliability Principles

standardadvanced

The reliability pillar of the Well-Architected Framework is AWS's written answer to "what does a reliable workload look like". It defines reliability as a workload performing its intended function correctly and consistently when expected to — including the ability to operate and test it through its whole lifecycle — and the practices behind that are the ones this section has been building toward.

Think of it as

The pillar is a checklist you read against a design rather than a technology to adopt. Its value is that it asks questions the designers of a system rarely ask themselves, in a form specific enough to answer yes or no.

text

# The six pillars, for context (§51 covers them in full)
operational excellence · security · reliability
performance efficiency · cost optimization · sustainability

Treating throttling as something to remove rather than something to design

Wrong

text

# "Customers are getting 429s — raise every limit until they stop."

Better

text

# Set the limit from what the system can actually serve, and return 429
# with Retry-After so callers back off correctly

What you see: The limits are raised, the 429s stop, and the service starts timing out instead — which is strictly worse, because now no request succeeds rather than most of them.

Why: Throttling is how a system protects the work it has already accepted. Removing the limit does not add capacity; it converts a clear, retryable rejection of some requests into degraded latency for all of them, and it removes the signal that would have told callers to slow down.

Each principle, as a question to ask about a design

Each principle, as a question to ask about a design
Principle	The question	A failing answer sounds like
Loosely coupled dependencies	Which of our dependencies can be down without us being down?	"All of them are required"
Throttling	What do we do with more work than we can handle?	"We accept it and hope"
Bounded retries	How many times, and with what backoff?	"Until it succeeds"
Fail fast	How long do we wait before giving up?	"There is no timeout"
Timeouts	Is every outbound call bounded?	"The defaults are probably fine"
Stateless services	What breaks if this instance disappears right now?	"The sessions on it"

Together

text

# Reliability, as a six-question review of any service
1. Which dependencies are optional, and do we degrade without them?
2. What is our behaviour above capacity?
3. Retry limit and backoff?
4. Timeout on every outbound call?
5. What is lost if one instance vanishes?
6. When did we last test the failover?

Remember: Reliability is a workload doing its job correctly and consistently, including being testable through its lifecycle. Loosen coupling, throttle deliberately, bound retries, fail fast, timeout everything, and keep services stateless — then review a design against those six as questions, not as advice.