beginnerintermediateadvanced

AWS · Distributed Tracing

Distributed Tracing and X-Ray / OpenTelemetry

Trace IDs and propagation, sampling and annotations, X-Ray vs OpenTelemetry, and reading a latency breakdown

One trace ID, many segmentsX-Amzn-Trace-Id: Root / Parent / Sampled
Traces are sampledfirst request/second + 5% by default
Annotations are indexed, metadata is notup to 50 annotations per trace
Round trip minus work = networkthe gap only exists if both ends are instrumented

1.Trace IDs, Segments, and Propagation

A trace follows one request through every service that handled it. Each service sends a segment describing its own work; a segment breaks down into subsegments for downstream calls it made. What ties them together is a trace ID carried in an HTTP header — `X-Amzn-Trace-Id` for X-Ray — added by the first traced service and passed along by everything after it.

Key features

X-Ray receives data as segments, groups segments sharing a trace ID into a trace, and processes traces into a service graph
A segment records the host, the request, the response, the timing, and any errors; subsegments break the work down into individual downstream calls
The tracing header is `X-Amzn-Trace-Id`, carrying `Root` (the trace ID), optionally `Parent` (the calling segment id), and `Sampled`
The first X-Ray-integrated service a request hits adds the tracing header; every service after it propagates the same trace ID
For services that do not send their own segments (DynamoDB, an external API), X-Ray builds an inferred segment from the caller's subsegment so the dependency still appears
A tracing header can come from the client, so an application should strip incoming `X-Amzn-Trace-Id` rather than trust a caller-supplied trace ID or sampling decision — trace and service graph data are retained for 30 days

The tracing header fields

The tracing header fields
Field	Meaning	Set by
Root	The trace ID that groups every segment for this request	The first X-Ray-integrated service the request hits
Parent	The segment id of the caller, so the child can be attached to it	An instrumented client making a downstream call
Sampled	1 or 0 — whether this request is being recorded	The sampling decision, made once at the front door
Lineage	Appended by Lambda and some services for their own processing	AWS services; not for your code to read or set

One trace ID across three services

Client

API

Orders service

DynamoDB

1POST /checkoutNo trace header yet
2Adds Root=1-5759e988-…;Sampled=1Segment: API
3POST /orders + same Root, Parent=API segment idSubsegment on API, segment on Orders
4PutItemSubsegment only — DynamoDB sends no segment, so X-Ray infers one
5200
6201

Signature

trace = segments (per service) + subsegments (per downstream call), grouped by Root trace ID

Segments (one per service) + subsegments (one per downstream call) grouped by a trace ID = a trace. `X-Amzn-Trace-Id` carries Root / Parent / Sampled, added at the front door and propagated onward — and every non-HTTP hop is somewhere you must carry it yourself.

2.Sampling, Annotations, and Service Maps

Tracing every request would cost more than it is worth, so X-Ray samples: by default it records the first request each second plus five percent of the rest. That makes traces a sample, not a log. Annotations are indexed key-value pairs you attach so you can find the traces you care about later; metadata is the same idea without the index. The service map is the aggregated view built from all of it.

Key features

The default sampling rule records the first request each second and five percent of any additional requests; you can add rules based on properties of the service or request
A sensible split is to trace all state-changing requests and sample high-volume read-only traffic (health checks, polling) at a low rate
Annotations are key-value pairs indexed for filter expressions — X-Ray indexes up to 50 annotations per trace
Metadata is key-value data of any type, including objects and lists, that is stored but not indexed and cannot be searched on
A group is a saved filter expression that gets its own service graph and publishes CloudWatch metrics every minute for traces matching it
The service graph node uses the downstream service's own segment when it exists while the edge uses the upstream's subsegment, and X-Ray classifies problems as Error (4xx), Fault (5xx), or Throttle (429)

Sampling rules worth writing

Sampling rules worth writing
Traffic	Rate	Why
Health checks, connection polling	Very low or none	High volume, near-zero diagnostic value
Read-only list endpoints	Low percentage	A sample is enough to see a latency regression
State-changing requests (payments, orders)	Trace everything	Each one is individually worth explaining
A newly deployed route	Temporarily high	You want detail while the behaviour is still unknown

Annotation or metadata?

Annotation — indexed

Simple key-value pairs only
Searchable with filter expressions
Up to 50 indexed per trace
For anything you will want to filter by: tenant, route, plan tier

Metadata — not indexed

Any type, including objects and lists
Stored and viewable, never searchable
For context you read once you have found the trace
Request bodies, computed values, feature-flag state

Signature

sampling (1/s + 5%) · annotation = indexed (≤50/trace) · metadata = stored, not indexed · group = saved filter + metrics

Traces are sampled — the default is the first request each second plus 5% of the rest — so they explain failures rather than count them. Annotations are indexed and searchable (50 per trace); metadata is stored but not. Groups turn a filter expression into its own service graph plus CloudWatch metrics.

3.X-Ray and OpenTelemetry

There are two ways to produce traces on AWS. The X-Ray SDK is AWS-native: it speaks X-Ray's own segment format and its own tracing header. OpenTelemetry is an open standard for instrumenting applications, and AWS accepts OpenTelemetry data directly at an OTLP endpoint. The instrumentation you write differs; the traces you end up reading are in the same console.

Key features

OpenTelemetry is an open-source observability framework providing a unified format for instrumenting, collecting, and exporting metrics, logs, and traces
AWS documents avoiding vendor lock-in as the reason to use it — the instrumentation is portable even though the backend is not
You can send traces directly to an OpenTelemetry Protocol (OTLP) endpoint and get application performance monitoring in CloudWatch Application Signals
CloudWatch also accepts OpenTelemetry metrics over OTLP, which use metric names with up to 150 labels instead of namespaces and up-to-30 dimensions
OpenTelemetry metrics are queried with PromQL in CloudWatch Query Studio, and support PromQL-based CloudWatch alarms
The X-Ray SDK remains the lowest-friction option when the whole stack is AWS-native and the tracing header is already propagated for you

Choosing an instrumentation path

Choosing an instrumentation path
Situation	Reach for	Why
All-AWS stack, tracing needed today	X-Ray SDK	Header propagation and AWS SDK instrumentation come built in
Hybrid or multi-cloud, or a second observability vendor in play	OpenTelemetry (OTLP)	One instrumentation, several possible backends
Existing OpenTelemetry instrumentation from another platform	OpenTelemetry (OTLP)	Keep the instrumentation, change only the exporter
You want Prometheus-style metrics and PromQL alarms	OpenTelemetry metrics over OTLP	CloudWatch queries these with PromQL in Query Studio

Command

text

# OpenTelemetry: instrument once, point the exporter at AWS
OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_ENDPOINT=<the AWS OTLP endpoint>
OTEL_TRACES_EXPORTER=otlp

X-Ray SDK = AWS-native, least setup on an all-AWS stack. OpenTelemetry = an open standard AWS accepts over OTLP, with CloudWatch Application Signals on the other side, and portability as the reason to pay its setup cost. Whichever you pick, one context format must own the whole request path.

4.Attributing Latency Across Tiers

A slow endpoint is not one number, it is a sum. Tracing splits that sum into named parts: time your own code spent computing, time waiting on a database, time waiting on another service, and time the request spent in transit. Each part has a different fix, so attributing the latency correctly is what stops you optimising the wrong thing.

Key features

AWS states both viewpoints are useful: the downstream service records when it started and ended work, and the upstream service records the round-trip latency including travel time between them
Round-trip time minus the downstream service's own duration is the network, queueing, and connection-acquisition overhead
A subsegment for a database call bounds database time; time inside the segment but outside every subsegment is your own code
Serial subsegments that could have been concurrent show up as a staircase in the timeline — the single most common avoidable pattern
X-Ray separates Error (4xx), Fault (5xx), and Throttle (429), so a slow-and-retrying downstream is distinguishable from a slow-but-succeeding one
The service map's edge carries the upstream's view and the node carries the service's own view, so a node that looks fast on an edge that looks slow is a network or queueing problem

Where the time went, and what to do about it

Where the time went, and what to do about it
Signature in the trace	What it means	The fix that helps
One long database subsegment	A slow query or a missing index	Fix the query; caching only hides it
Many short database subsegments in series	N+1 queries	Batch or join — more database capacity will not help
Round trip much larger than the downstream segment	Network, queueing, or connection acquisition	Connection pooling, keep-alive, co-location
Serial subsegments with no dependency between them	Missed concurrency	Issue the calls in parallel
Downstream throttles then retries	A quota, not slowness	Back off, or raise the quota
Large unaccounted time inside the segment	Your own CPU or blocking work	Profile the handler; tracing has taken you as far as it can

Reading a 4.2s trace

Total segment: 4,200 ms

What the client experienced

Subsegment: auth service 180 ms

Its own segment says 120 ms — 60 ms is network + queueing

Subsegment: SQL query 3,900 ms

One query, no concurrency — this is the incident

Unaccounted: 120 ms

Inside the segment, outside every subsegment — your own code

Signature

total = own code + db subsegments + downstream subsegments + (round trip − downstream duration)

Split the total into: your own code (inside the segment, outside every subsegment), database time, downstream time, and the gap between the caller's round trip and the callee's own duration — which is network and queueing. Each part has a different fix.

Spotted a mistake on this page? Tell us.

Read Sheet