beginnerintermediateadvanced

AWS · Section 59

Production Debugging on AWS

Level: advanced
Read: 40 min
Concepts: 3

Under pressure people debug the part of the system they know best, which is why a fixed order is worth more than any single tool. State the symptom as a measurement — which endpoint, what error, since when, what fraction, and crucially what is not affected — then check what changed recently, then walk outward through metrics, logs, traces, dependency health, network, IAM, quotas and finally cost signals. The order is chosen by how much each step eliminates per minute: metrics say where and when, logs say what, traces say which hop, and stopping as soon as the symptom is fully explained is part of the method. Underneath the method sits a catalogue, because AWS incidents repeat: a 5xx spike that is either the load balancer's or the application's depending on which metric moved, a latency spike a trace attributes to one hop, Lambda throttling that is a concurrency question rather than a code question, ECS restart loops from health checks or memory limits, RDS connection exhaustion with an idle database, an SQS backlog whose real question is whether consumers are succeeding at all, NAT port exhaustion that presents as one third party timing out, DNS and IAM and security-group failures with their own distinct signatures. Both the method and the catalogue depend on being able to find one request's worth of evidence, which is what a correlation ID provides: created at the entry point, bound into the logging context so every line carries it, propagated across HTTP calls and queue messages, and returned to the client on error. Traces apportion time but are sampled; correlation IDs cover every request. Using each for what it is good at is what keeps an investigation to minutes.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

A fixed order turns debugging into a search that narrows, rather than a series of guesses.
Naming what is unaffected eliminates more of the system than any single check.
Eleven signatures — a metric, an error and a first check each — cover most incidents.
A correlation ID in every log line is what makes logs joinable across services and queues.
Traces are sampled and logs are not, so they answer different questions.

What you will be able to do

Run an investigation in a fixed order and know when to stop
Recognise the eleven recurring AWS incident signatures on sight
Separate load-balancer errors from application errors in one look
Diagnose NAT port exhaustion that presents as a third-party fault
Design logging so one user report becomes one query across every service

A report becomes a cause

A user-visible failure

vague, urgent, partial

Follow the fixed order

symptom → change → metrics → logs → traces →…

Recognise the signature

one of eleven recurring shapes

Search by correlation id

every line, every service, sync and async

One hop, one cause

named, with evidence and a timestamp

Production Debugging on AWS

The investigation order that narrows the search, the eleven signatures worth recognising instantly, and the identifiers that let one request be reconstructed across every service it touched.

The Fixed Investigation Order

coreadvanced

Under pressure, people debug by guessing at the part of the system they know best. A fixed order removes the guessing: state the symptom precisely, check what changed recently, then walk outward through metrics, logs, traces, dependencies, network, permissions and quotas. Each step either explains the symptom or eliminates a whole region of the system.

Think of it as

Debugging is a search, and the order is chosen by how much each check eliminates per minute. "What changed?" comes second because it explains most incidents in one question. Quotas and IAM come late not because they are rare but because they produce specific errors, so by the time you reach them you already know where to look.

What we're doing: Work one incident through the order and watch the search space collapse.

incident-walk.txttext

1 SYMPTOM   Checkout p95 went from 240 ms to 9 s at 09:12.
            Error rate normal. Nothing is failing — it is slow.
            → "slow, not failing" removes every error-path theory.

2 CHANGE    Last deploy was 40 hours ago. No config change.
            One thing did change: a scheduled report job was
            enabled this morning, first run 09:10.
            → Two minutes before the symptom. Held as a candidate.

3 METRICS   Web tier CPU flat. RDS CPU 95%, read IOPS tripled,
            connections at the pool maximum.
            → The web tier is waiting, not working. Move down.

4 LOGS      No errors. Handlers start and complete, slowly.
            → Confirms a wait, not a fault. Logs are exhausted
              as a source; do not keep reading them.

5 TRACES    p95 trace: 8.6 s of 9 s inside one database segment,
            a single query on the orders table.
            → One query. Not the application, not the network.

6 DEPENDENCY  Performance Insights shows that query as the top
              consumer, running from the report job's connection.
              → Cause confirmed, two minutes of evidence apart.

STOP. Steps 7 to 10 are not run, because the symptom is fully
explained. The fix is to move the report to a read replica.

1: "Slow but not failing" is a different investigation from "failing", and separating them at step 1 saves the most time of any decision here.
6: Step 2 rarely proves anything on its own, but it produces the candidate that later steps confirm or discard.
12: Flat CPU on the tier that looks slow is the signal to stop looking at it — saturation is somewhere downstream.
18: Traces answer the "which hop" question directly, which is why they sit above dependency inspection rather than below it.
23: Stopping is part of the method. Continuing past a full explanation is how one incident becomes three theories.

Why this works: The order is not sacred, but having one is. Six steps ran, four were unnecessary, and every step either eliminated a region of the system or produced the next candidate. The same incident debugged by intuition usually starts at logs, finds nothing because there are no errors, and loses twenty minutes before anyone looks at the database.

Opening the logs first

Wrong

text

# "Something is broken" → tail the application logs and
# read until something looks wrong.

Better

text

# Symptom → recent change → metrics.
# Only then open the logs, filtered to the failing requests.

What you see: Twenty minutes reading normal log lines from a service that turns out to be healthy, while the actual failure is one tier away.

Why: Logs are high volume and low selectivity: they tell you what happened in one component, but not which component to read. Metrics answer "where and when" in seconds and turn the log search from an open-ended read into a filtered query over a known time window.

Ten steps, narrowing at every one

The funnel is the point: each step removes candidates rather than confirming a hunch. Skipping to a favourite step widens the search instead of narrowing it.

Each step: the question, where to look, and what a hit looks like

Each step: the question, where to look, and what a hit looks like
Step	The question	Where	A hit looks like
1 Symptom	What exactly is failing, and since when?	The report, plus a dashboard	A precise statement: "POST /orders, 502, 12% since 14:05"
2 Recent change	What changed near that time?	Deployment history, CloudTrail, Config timeline	A change within minutes of the start time
3 Metrics	Where in the system, and when exactly?	CloudWatch metrics and alarms	One tier's error or latency curve matching the symptom
4 Logs	What did the failing requests say?	CloudWatch Logs, filtered by the request id	A recurring error, or a handler that starts and never finishes
5 Traces	Which hop consumed the time?	X-Ray or OpenTelemetry trace map	One segment holding most of the latency
6 Dependencies	Is something we depend on unhealthy?	RDS, ElastiCache, queue depth, provider status	Connections at maximum, or a queue growing without draining
7 Network	Can the packets get there?	Security groups, NACLs, route tables, Flow Logs, DNS	Rejected connections, or a name resolving to the wrong target
8 IAM	Are we allowed to do this?	Error message, CloudTrail	AccessDenied naming a principal, action and resource
9 Quotas	Have we hit a ceiling?	Service Quotas utilization, throttling metrics	Throttling or a limit-exceeded error at a round number
10 Cost	Is spend telling us something the metrics are not?	Cost Explorer, anomaly detection	A cost or usage curve that turns at the same minute

Together

text

# Write step 1 down before touching anything else
SYMPTOM   POST /orders returns 502
SCOPE     ~12% of requests, all Regions, all clients
SINCE     14:05 UTC, ongoing
NOT       GET endpoints are unaffected; the site loads normally

# The "NOT" line is the most valuable one: it has already
# eliminated DNS, TLS, the CDN and the load balancer itself.

Remember: Symptom, recent change, metrics, logs, traces, dependencies, network, IAM, quotas, cost — in that order. Write the symptom down including what is not affected, because that line eliminates more of the system than anything else you will do, and stop as soon as the symptom is fully explained.

The Eleven Investigations You Will Actually Run

coreadvanced

Production incidents on AWS repeat. The same eleven investigations account for most of them, and each has a signature — a specific metric, a specific error, a first place to look. Recognising the signature is what turns a twenty-minute search into a two-minute confirmation.

Think of it as

Each investigation is a metric plus an error message plus a first check. Learn the triple rather than the story: when you see `ErrorPortAllocation` you are already looking at NAT exhaustion, and when you see connections at the pool maximum with a healthy database you are already looking at a pool that is not returning connections.

What we're doing: Diagnose a NAT gateway exhaustion incident that presents as random third-party timeouts.

nat-exhaustion.txttext

SYMPTOM  Calls to one payment provider time out for ~3% of
         requests, at random, only during the daily peak. Every
         other outbound call is fine, including to other providers.

WHY IT LOOKS LIKE THE PROVIDER  Errors are timeouts, they are
intermittent, and they name a third party. The first four hours
of this incident are usually spent emailing the provider.

THE SIGNATURE  The failures are per-destination, not per-service.
CloudWatch on the NAT gateway shows ErrorPortAllocation > 0
during exactly the failing windows.

WHY  A NAT gateway supports up to 55,000 simultaneous connections
to a single destination per IPv4 address. The payment integration
opens a new connection per request rather than reusing a pool, so
at peak the count against that one destination crosses the limit.
Connections to every other destination are unaffected — which is
what made it look like one provider was at fault.

FIXES, in order of how much they help
  1. Reuse connections in the client (a pool with keep-alive)
  2. Add secondary IP addresses to the NAT gateway — each adds
     another 55,000 concurrent connections to that destination
  3. One NAT gateway per AZ, with clients spread across zones
  4. Close idle connections; watch IdleTimeoutCount

1: "Only one destination, only at peak" is the whole diagnosis — the shape of the symptom names the limit.
7: Naming why an investigation goes wrong is as useful as naming what is right; this one wastes hours on a healthy vendor.
14: The limit is per destination per IP address, which is exactly why other traffic through the same gateway is unaffected.
20: Fixing the client first is right: adding IPs raises the ceiling, and connection reuse removes the pressure.

Why this works: This investigation is worth knowing in detail because every part of it is misleading. The errors point at a third party, the gateway looks healthy, no service metric moves, and the limit involved is one most people have never read. The `ErrorPortAllocation` metric turns all of that into a single check.

Scaling out the workers when the queue backlog grows

Wrong

text

# ApproximateNumberOfMessagesVisible climbing → raise desired
# count from 10 to 60 and wait.

Better

text

# First: are consumers succeeding at all?
# Check NumberOfMessagesDeleted and the DLQ depth. If deletions
# are zero, more consumers will not help — they will all fail too.

What you see: Sixty workers now fail on the same message, the dead-letter queue fills six times faster, and the downstream service being hammered by retries gets worse.

Why: A backlog has two causes that look identical on the depth metric: not enough capacity, and consumers that are not succeeding. The deletion rate separates them in one look. Scaling into the second case multiplies the failure rate and can take down whatever the consumers depend on.

Eleven signatures, grouped by where they live

It is returning errors

5xx spike

ALB 5xx vs target 5xx tells you who produced it

IAM AccessDenied

the message names principal, action, resource

Security group refusal

connection times out rather than refusing

It is slow, not failing

Latency spike

a trace attributes it to one hop

RDS connection exhaustion

pool at maximum, database idle

DNS failure

partial, cache-dependent, looks intermittent

It is at a ceiling

Lambda throttles

429 from Lambda — concurrency, not code

EC2 CPU saturation

or credit exhaustion on a burstable type

NAT port exhaustion

55,000 per destination per IP

It is falling behind

SQS backlog

age of oldest message, not depth

ECS restart loop

health check, memory limit, or startup exit

Signature, first check, and the cause it usually is

Signature, first check, and the cause it usually is
Investigation	Signature	First check	Usual cause
5xx spike	ALB HTTPCode_ELB_5XX vs HTTPCode_Target_5XX	Which of the two moved	ELB 5xx means no healthy target; target 5xx means the app returned it
Latency spike	p95 up, error rate flat	A trace map for the slow period	One downstream hop — usually a query or a third party
Lambda throttles	429 from Lambda, `Throttles` metric > 0	Concurrent executions against the account quota	Reserved concurrency too low, or the Region quota reached
ECS restart loop	Tasks stopping minutes after starting	The stopped-task reason string	Failed health check, OOM kill, or a non-zero exit at startup
EC2 CPU saturation	CPUUtilization pinned, or CPU credits at zero	Whether the instance type is burstable	Undersized instance, or a burstable type used for steady load
RDS connection exhaustion	DatabaseConnections at max, low CPU	Connections per task × task count	Pools not returning connections, or a scale-out multiplying pools
SQS backlog	ApproximateAgeOfOldestMessage climbing	Whether consumers are running and succeeding	Workers crashed, scaled on the wrong metric, or poisoned by one message
NAT exhaustion / cost	`ErrorPortAllocation` > 0, or a large NAT data bill	Destination distribution and `IdleTimeoutCount`	Many connections to one destination; or S3 traffic with no gateway endpoint
DNS failure	Intermittent, differs per client, resolves eventually	The record itself, and its TTL	A record change still propagating, or resolver configuration
IAM AccessDenied	Explicit error naming action and resource	The message, then CloudTrail for the same call	A missing action, an SCP, or a resource policy
Security group issue	Connection hangs then times out	Whether the rule references the caller's group	A rule referencing a CIDR that changed, or a missing return path on a NACL

Together

text

# The distinction that resolves most 5xx investigations in one look
HTTPCode_ELB_5XX_Count     > 0  → the load balancer produced it.
                                  No healthy target, or the target
                                  did not respond in time.
HTTPCode_Target_5XX_Count  > 0  → the application produced it.
                                  Read the application logs.

# Both non-zero means both are happening, which is a different
# incident again — usually saturation causing timeouts.

Remember: Eleven signatures cover most AWS incidents. Split ALB 5xx from target 5xx before anything else, read the AccessDenied message rather than widening a policy, check whether consumers are succeeding before scaling a backlog, and remember that NAT allows 55,000 simultaneous connections per destination per IP address.

Correlation IDs and Request IDs

coreadvanced

A correlation ID is one value attached to a request at its entry point and carried through every service, log line and downstream call it touches. It turns "a customer says checkout failed at about two o'clock" into one exact search that returns every log line from every service for that one request.

Think of it as

There are two different identifiers and they answer different questions. A correlation ID is yours: it ties a single user action together across your services. A service request ID is AWS's: it identifies one call to one AWS API and is what Support needs. Log both, and one incident stays searchable from either end.

What we're doing: Turn a vague customer report into a complete picture of one request.

from-report-to-cause.txttext

WITHOUT a correlation id
  "Checkout failed around 2pm." Search the API logs for 500s in a
  20-minute window: 340 of them. Guess which is theirs by user id
  — if the user id is even logged. Then repeat the guess in the
  worker logs, which have no user id at all.

WITH a correlation id, returned in the error response
  The user quotes c-9f13ab77 from the error page. One query:

    fields @timestamp, service, msg
    | filter correlation_id = "c-9f13ab77"
    | sort @timestamp asc

  Returns, in order, across four log groups:
    14:07:21  orders-api   checkout started
    14:07:21  orders-api   inventory reserved
    14:07:22  orders-api   payment call timed out after 800ms
    14:07:22  orders-api   returned 500
    14:09:04  reconciler   released stale inventory reservation

  The last line is the one nobody would have found: the cleanup
  ran two minutes later in a different service, and it is part of
  the same user action.

WITH the trace id as well
  The trace shows 780 ms of the 800 ms inside the payment
  provider's segment. Not a network problem, not our code.

1: The 340-candidate search is the normal experience, and it is why "we have logs" is not the same as "we can investigate".
6: Returning the id to the client is a small change that moves the search key from your side to the user's.
13: The asynchronous line is the payoff: the id crossed a queue boundary that a trace or a request id would not have followed.
19: Correlation ids find the lines; traces apportion the time. Both, together, name the cause.

Why this works: Every service logs plenty; almost none of it is joinable. One field, present in every line and propagated across every boundary including queues, converts a pile of logs into a queryable record of what happened. It costs one middleware and one convention, and it is the difference between a two-minute investigation and an afternoon.

Logging the correlation ID only where the request enters

Wrong

text

logger.info("request received", extra={"correlation_id": cid})
# ...every later line in the request logs without it.

Better

text

# Bind it once per request in a context variable, and have the
# formatter add it to every line automatically — including
# library and framework logs.

What you see: A search for the id returns exactly one line: the arrival. The error itself, three functions deeper, is unfindable by that id.

Why: The value of a correlation ID is proportional to how many lines carry it. Adding it by hand means it is present exactly where someone remembered, which is usually the paths that were already easy to debug. Binding it in the logging context makes coverage the default and forgetting it impossible.

One id, attached once, carried everywhere

Browser

ALB

API

Worker

CloudWatch

1. POST /checkoutNo id yet — this is where one is created
2. forward, with X-Amzn-Trace-IdThe edge adds the tracing header
3. log {correlation_id, trace_id, user_id, route}One JSON line per event, always carrying the id
4. enqueue job, id in the message attributesAsync work keeps the same correlation id
5. 500, body includes the correlation idThe user can now quote the search key
6. log {correlation_id, …} for every stepSame id, hours later, different service
7. one query returns every line for that requestAcross services, synchronous and asynchronous alike

Which identifier answers which question

Which identifier answers which question
Identifier	Created by	Scope	Use it to
Correlation ID	Your entry point (API, or the client)	The whole user action, including asynchronous follow-up	Find every log line for one user action across every service
Trace ID (`X-Amzn-Trace-Id`)	The first X-Ray-integrated service reached	One traced request path	See where the time went, hop by hop
AWS request ID	The AWS service being called	One AWS API call	Quote to AWS Support; correlate with CloudTrail
ALB request ID	The load balancer	One request through the ALB	Join access logs to application logs
Message ID / receipt handle	SQS	One message	Trace one delivery, not one user action

Together

text

# One structured log line — every field earns its place
{"ts":"2026-09-04T14:07:22Z","level":"error",
 "correlation_id":"c-9f13ab77","trace_id":"1-68b9a1f2-4c2e…",
 "service":"orders-api","route":"POST /checkout",
 "user_id":"u-4471","tenant":"t-14",
 "aws_request_id":"8b1a2c44-…","msg":"payment call timed out"}

# correlation_id  → every service, this whole action
# trace_id        → the latency breakdown for this hop chain
# aws_request_id  → the single AWS API call, for Support

Remember: Create one correlation ID at the entry point, bind it into the logging context so every line carries it, propagate it across HTTP calls and queue messages, and return it to the client on error. Log the trace ID and the AWS request ID beside it — traces apportion time, correlation IDs find the lines, and sampling means traces alone will miss requests.