Production Debugging on AWS cheat sheet · AWS

1.The Fixed Investigation Order

Under pressure, people debug by guessing at the part of the system they know best. A fixed order removes the guessing: state the symptom precisely, check what changed recently, then walk outward through metrics, logs, traces, dependencies, network, permissions and quotas. Each step either explains the symptom or eliminates a whole region of the system.

Key features

State the symptom as a measurement — which endpoint, what error, since when, what fraction of requests — before opening any console
Most incidents follow a change: a deploy, a configuration edit, a feature flag, a scaling event, or a dependency's own release
Metrics tell you where and when; logs tell you what; traces tell you which hop — reaching for logs first means reading the wrong service's logs
A trace shows application time, database time and downstream time separately, which is the question "why is it slow" actually asks
IAM failures are specific and loud: an AccessDenied names the principal, the action and the resource
A cost or usage anomaly that starts at the same minute as the symptom is evidence, not a coincidence

Each step: the question, where to look, and what a hit looks like

Each step: the question, where to look, and what a hit looks like
Step	The question	Where	A hit looks like
1 Symptom	What exactly is failing, and since when?	The report, plus a dashboard	A precise statement: "POST /orders, 502, 12% since 14:05"
2 Recent change	What changed near that time?	Deployment history, CloudTrail, Config timeline	A change within minutes of the start time
3 Metrics	Where in the system, and when exactly?	CloudWatch metrics and alarms	One tier's error or latency curve matching the symptom
4 Logs	What did the failing requests say?	CloudWatch Logs, filtered by the request id	A recurring error, or a handler that starts and never finishes
5 Traces	Which hop consumed the time?	X-Ray or OpenTelemetry trace map	One segment holding most of the latency
6 Dependencies	Is something we depend on unhealthy?	RDS, ElastiCache, queue depth, provider status	Connections at maximum, or a queue growing without draining
7 Network	Can the packets get there?	Security groups, NACLs, route tables, Flow Logs, DNS	Rejected connections, or a name resolving to the wrong target
8 IAM	Are we allowed to do this?	Error message, CloudTrail	AccessDenied naming a principal, action and resource
9 Quotas	Have we hit a ceiling?	Service Quotas utilization, throttling metrics	Throttling or a limit-exceeded error at a round number
10 Cost	Is spend telling us something the metrics are not?	Cost Explorer, anomaly detection	A cost or usage curve that turns at the same minute

Ten steps, narrowing at every one

The funnel is the point: each step removes candidates rather than confirming a hunch. Skipping to a favourite step widens the search instead of narrowing it.

Signature

symptom → recent change → metrics → logs → traces → dependencies → network → IAM → quotas → cost

Symptom, recent change, metrics, logs, traces, dependencies, network, IAM, quotas, cost — in that order. Write the symptom down including what is not affected, because that line eliminates more of the system than anything else you will do, and stop as soon as the symptom is fully explained.

2.The Eleven Investigations You Will Actually Run

Production incidents on AWS repeat. The same eleven investigations account for most of them, and each has a signature — a specific metric, a specific error, a first place to look. Recognising the signature is what turns a twenty-minute search into a two-minute confirmation.

Key features

A NAT gateway supports up to 55,000 simultaneous connections per destination per IPv4 address; each additional IP address adds another 55,000
A NAT gateway connection idle for 350 seconds or more times out, and the gateway sends an RST rather than a FIN
The `ErrorPortAllocation` and `IdleTimeoutCount` metrics are the direct evidence for NAT port exhaustion
Lambda throttling is a 429 from Lambda itself and is a concurrency question, not a code question
An ECS task restart loop is usually a failing health check, an exhausted memory limit, or a container exiting non-zero at startup
AccessDenied names the principal, the action and the resource — read the message rather than guessing at the policy

Signature, first check, and the cause it usually is

Signature, first check, and the cause it usually is
Investigation	Signature	First check	Usual cause
5xx spike	ALB HTTPCode_ELB_5XX vs HTTPCode_Target_5XX	Which of the two moved	ELB 5xx means no healthy target; target 5xx means the app returned it
Latency spike	p95 up, error rate flat	A trace map for the slow period	One downstream hop — usually a query or a third party
Lambda throttles	429 from Lambda, `Throttles` metric > 0	Concurrent executions against the account quota	Reserved concurrency too low, or the Region quota reached
ECS restart loop	Tasks stopping minutes after starting	The stopped-task reason string	Failed health check, OOM kill, or a non-zero exit at startup
EC2 CPU saturation	CPUUtilization pinned, or CPU credits at zero	Whether the instance type is burstable	Undersized instance, or a burstable type used for steady load
RDS connection exhaustion	DatabaseConnections at max, low CPU	Connections per task × task count	Pools not returning connections, or a scale-out multiplying pools
SQS backlog	ApproximateAgeOfOldestMessage climbing	Whether consumers are running and succeeding	Workers crashed, scaled on the wrong metric, or poisoned by one message
NAT exhaustion / cost	`ErrorPortAllocation` > 0, or a large NAT data bill	Destination distribution and `IdleTimeoutCount`	Many connections to one destination; or S3 traffic with no gateway endpoint
DNS failure	Intermittent, differs per client, resolves eventually	The record itself, and its TTL	A record change still propagating, or resolver configuration
IAM AccessDenied	Explicit error naming action and resource	The message, then CloudTrail for the same call	A missing action, an SCP, or a resource policy
Security group issue	Connection hangs then times out	Whether the rule references the caller's group	A rule referencing a CIDR that changed, or a missing return path on a NACL

Eleven signatures, grouped by where they live

It is returning errors

5xx spike

ALB 5xx vs target 5xx tells you who produced it

IAM AccessDenied

the message names principal, action, resource

Security group refusal

connection times out rather than refusing

It is slow, not failing

Latency spike

a trace attributes it to one hop

RDS connection exhaustion

pool at maximum, database idle

DNS failure

partial, cache-dependent, looks intermittent

It is at a ceiling

Lambda throttles

429 from Lambda — concurrency, not code

EC2 CPU saturation

or credit exhaustion on a burstable type

NAT port exhaustion

55,000 per destination per IP

It is falling behind

SQS backlog

age of oldest message, not depth

ECS restart loop

health check, memory limit, or startup exit

Signature

5xx · latency · Lambda throttles · ECS restarts · CPU · RDS connections · SQS backlog · NAT ports · DNS · IAM · SG

Eleven signatures cover most AWS incidents. Split ALB 5xx from target 5xx before anything else, read the AccessDenied message rather than widening a policy, check whether consumers are succeeding before scaling a backlog, and remember that NAT allows 55,000 simultaneous connections per destination per IP address.

3.Correlation IDs and Request IDs

A correlation ID is one value attached to a request at its entry point and carried through every service, log line and downstream call it touches. It turns "a customer says checkout failed at about two o'clock" into one exact search that returns every log line from every service for that one request.

Key features

X-Ray propagates the trace with an `X-Amzn-Trace-Id` header carrying a root trace ID, an optional parent segment ID, and the sampling decision
The first X-Ray-integrated service the request reaches adds the tracing header, and downstream services read and extend it
X-Ray samples by default — the SDK records the first request each second and five percent of additional requests — so not every request has a trace
Annotations are indexed key-value pairs you can filter traces by, up to 50 per trace; metadata is not indexed and is for context you only read
Your correlation ID belongs in every structured log line, not only in the entry-point log, or the search only finds one service
Return the correlation ID to the client in the error response, so a user report arrives with the search key already attached

Which identifier answers which question

Which identifier answers which question
Identifier	Created by	Scope	Use it to
Correlation ID	Your entry point (API, or the client)	The whole user action, including asynchronous follow-up	Find every log line for one user action across every service
Trace ID (`X-Amzn-Trace-Id`)	The first X-Ray-integrated service reached	One traced request path	See where the time went, hop by hop
AWS request ID	The AWS service being called	One AWS API call	Quote to AWS Support; correlate with CloudTrail
ALB request ID	The load balancer	One request through the ALB	Join access logs to application logs
Message ID / receipt handle	SQS	One message	Trace one delivery, not one user action

One id, attached once, carried everywhere

Browser

ALB

API

Worker

CloudWatch

1POST /checkoutNo id yet — this is where one is created
2forward, with X-Amzn-Trace-IdThe edge adds the tracing header
3log {correlation_id, trace_id, user_id, route}One JSON line per event, always carrying the id
4enqueue job, id in the message attributesAsync work keeps the same correlation id
5500, body includes the correlation idThe user can now quote the search key
6log {correlation_id, …} for every stepSame id, hours later, different service
7one query returns every line for that requestAcross services, synchronous and asynchronous alike

Signature

correlation_id (yours, every line, crosses queues) · X-Amzn-Trace-Id (X-Ray, sampled) · AWS request id (one API call)

Create one correlation ID at the entry point, bind it into the logging context so every line carries it, propagate it across HTTP calls and queue messages, and return it to the client on error. Log the trace ID and the AWS request ID beside it — traces apportion time, correlation IDs find the lines, and sampling means traces alone will miss requests.