beginnerintermediateadvanced

AWS · Section 38

Distributed Tracing and X-Ray / OpenTelemetry

Level: advanced
Read: 35 min
Concepts: 4

A trace follows one request across every service that handled it. Each service emits a segment describing its own work, broken into subsegments for the downstream calls it made, and a trace ID carried in the `X-Amzn-Trace-Id` header is what groups them. The first traced service mints that header; everything after it propagates the same Root — which happens automatically over instrumented HTTP clients and never happens over a queue unless you carry it yourself. Tracing is sampled (the default records the first request each second plus five percent of the rest), so a trace corpus explains failures rather than counting them, and annotations — indexed, up to fifty per trace — are what make specific traces findable later. Instrumentation is either the AWS-native X-Ray SDK or OpenTelemetry sent to an OTLP endpoint, the latter costing more setup and buying portability. The payoff is attribution: comparing an upstream subsegment against the downstream service's own segment separates work from network and queueing, and time inside a segment but outside every subsegment is your own code — three parts of one latency number with three different fixes.

This section

How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service LensFooBar Serverless

What is true here

Segments (per service) plus subsegments (per downstream call), grouped by a Root trace ID, make one trace.
Propagation is automatic over instrumented HTTP and manual over queues, streams, and scheduled jobs.
Sampling defaults to the first request per second plus 5% — traces explain, metrics count.
Annotations are indexed and searchable (50 per trace); metadata is stored but not indexed.
Round-trip time minus the downstream service's own duration is network, queueing, and connection acquisition.

What you will be able to do

Explain how a trace ID propagates and where propagation silently breaks
Write sampling rules that trace what matters without paying to trace health checks
Choose between annotations and metadata, and find traces with a filter expression
Decide between the X-Ray SDK and OpenTelemetry for a given stack
Read a trace and attribute latency to application code, database, downstream services, or the network

From a propagated trace ID to an explained latency number

Trace IDs, segments, propagation

Sampling, annotations, service maps

Where the time actually went

Distributed Tracing and X-Ray / OpenTelemetry

The trace data model and propagation, sampling and annotations, the instrumentation choice, and using a trace to attribute latency.

Trace IDs, Segments, and Propagation

coreintermediate

A trace follows one request through every service that handled it. Each service sends a segment describing its own work; a segment breaks down into subsegments for downstream calls it made. What ties them together is a trace ID carried in an HTTP header — `X-Amzn-Trace-Id` for X-Ray — added by the first traced service and passed along by everything after it.

Think of it as

A trace is a parcel-tracking number. Each depot scans the same number and records when it received and forwarded the parcel. Nobody has the whole journey; the number is what lets you assemble it afterwards. Lose the number at one depot and the journey splits into two unrelated deliveries.

What we're doing: See why a trace stops halfway through and the second half appears as a separate, parentless trace.

broken-propagation.txttext

API service: instrumented, adds Root=1-5759e988-... to the request.

Orders service: instrumented, receives the header, records its segment
under the same Root. Good.

Worker: reads a job off SQS and calls the pricing service. The job body
carries the business payload only — nobody put the trace ID in it.

Result: the pricing call starts a brand new trace. The service map shows
two disconnected graphs, and the checkout latency breakdown stops at the
queue.

4: Propagation over synchronous HTTP is usually handled for you by the instrumented HTTP client — this half works without anyone thinking about it.
7: An asynchronous hop has no HTTP headers to carry the context. The trace ID has to be put into the message and read back out by the consumer, deliberately.

Why this works: Tracing looks automatic because the SDK instruments HTTP clients for you. Every hop that is not an HTTP call — a queue, a stream, a scheduled job — is a place where the trace context has to be carried by hand, and where it silently is not.

Trusting an incoming X-Amzn-Trace-Id header on a public endpoint

Wrong

text

# Pass the client's X-Amzn-Trace-Id straight through

Better

text

# Strip X-Amzn-Trace-Id from incoming public requests and let the front
# door mint a fresh trace ID and sampling decision

What you see: Traces from unrelated requests merge together, or every request arrives already marked `Sampled=1` and the tracing bill rises far above the configured sampling rate.

Why: AWS documents this directly: a tracing header can originate from the client request, so an application should remove `X-Amzn-Trace-Id` from incoming requests to avoid users adding trace IDs or sampling decisions of their own. The sampling decision in particular controls spend.

One trace ID across three services

Client

API

Orders service

DynamoDB

1. POST /checkoutNo trace header yet
2. Adds Root=1-5759e988-…;Sampled=1Segment: API
3. POST /orders + same Root, Parent=API segment idSubsegment on API, segment on Orders
4. PutItemSubsegment only — DynamoDB sends no segment, so X-Ray infers one
5. 200
6. 201

The tracing header fields

The tracing header fields
Field	Meaning	Set by
Root	The trace ID that groups every segment for this request	The first X-Ray-integrated service the request hits
Parent	The segment id of the caller, so the child can be attached to it	An instrumented client making a downstream call
Sampled	1 or 0 — whether this request is being recorded	The sampling decision, made once at the front door
Lineage	Appended by Lambda and some services for their own processing	AWS services; not for your code to read or set

Together

text

X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1

Remember: Segments (one per service) + subsegments (one per downstream call) grouped by a trace ID = a trace. `X-Amzn-Trace-Id` carries Root / Parent / Sampled, added at the front door and propagated onward — and every non-HTTP hop is somewhere you must carry it yourself.

Sampling, Annotations, and Service Maps

coreintermediate

Tracing every request would cost more than it is worth, so X-Ray samples: by default it records the first request each second plus five percent of the rest. That makes traces a sample, not a log. Annotations are indexed key-value pairs you attach so you can find the traces you care about later; metadata is the same idea without the index. The service map is the aggregated view built from all of it.

Think of it as

Sampling turns tracing from a recording into a survey. A survey answers "what does a typical slow request look like" very well and "what happened to this one customer's order" not at all — which is why annotations exist, and why logs still carry the per-request detail.

What we're doing: Find the traces for one customer's failing checkouts among millions of requests.

annotate-then-filter.txttext

In the checkout handler:
  segment.put_annotation("tenant_id", tenant_id)
  segment.put_annotation("plan", "enterprise")
  segment.put_metadata("cart", cart_dict)   # context, not searchable

Filter expression in the console:
  annotation.tenant_id = "acme-4471" AND fault

-> the subset of sampled traces for that tenant that ended in a 5xx.

1: Annotations have to be added while the request is being served. A trace that was already recorded cannot be annotated afterwards.
5: `fault` is X-Ray's own classification for 5xx, distinct from `error` (4xx) and `throttle` (429) — filtering on it separates our failures from the caller's.

Why this works: Without annotations, a trace corpus is only searchable by URL and time, which is enough to spot a general regression and useless for a specific complaint. Annotations are the difference, and the 50-per-trace index limit is why they are for dimensions you filter on rather than for every value you have.

Putting a high-cardinality value in an annotation

Wrong

text

segment.put_annotation("request_body", json.dumps(body))

Better

text

segment.put_annotation("route", "/checkout")      # low cardinality
segment.put_metadata("request_body", body)        # not indexed

What you see: The indexed annotation budget is consumed by values nobody ever filters on, and the annotations that would actually be useful cannot be added.

Why: X-Ray indexes up to 50 annotations per trace, and an index is only worth its cost for values you group or filter by. Anything you would read rather than search on belongs in metadata, which accepts objects and lists and does not consume the index.

Annotation or metadata?

Annotation — indexed

+Simple key-value pairs only
+Searchable with filter expressions
+Up to 50 indexed per trace
+For anything you will want to filter by: tenant, route, plan tier

Metadata — not indexed

−Any type, including objects and lists
−Stored and viewable, never searchable
−For context you read once you have found the trace
−Request bodies, computed values, feature-flag state

Sampling rules worth writing

Sampling rules worth writing
Traffic	Rate	Why
Health checks, connection polling	Very low or none	High volume, near-zero diagnostic value
Read-only list endpoints	Low percentage	A sample is enough to see a latency regression
State-changing requests (payments, orders)	Trace everything	Each one is individually worth explaining
A newly deployed route	Temporarily high	You want detail while the behaviour is still unknown

Together

text

# A rule that traces every checkout, ahead of the default rule
{
  "RuleName": "trace-all-checkouts",
  "Priority": 100,
  "FixedRate": 1.0,
  "ReservoirSize": 0,
  "ServiceName": "api",
  "HTTPMethod": "POST",
  "URLPath": "/checkout*",
  "Host": "*", "ServiceType": "*", "ResourceARN": "*", "Version": 1
}

Remember: Traces are sampled — the default is the first request each second plus 5% of the rest — so they explain failures rather than count them. Annotations are indexed and searchable (50 per trace); metadata is stored but not. Groups turn a filter expression into its own service graph plus CloudWatch metrics.

X-Ray and OpenTelemetry

standardintermediate

There are two ways to produce traces on AWS. The X-Ray SDK is AWS-native: it speaks X-Ray's own segment format and its own tracing header. OpenTelemetry is an open standard for instrumenting applications, and AWS accepts OpenTelemetry data directly at an OTLP endpoint. The instrumentation you write differs; the traces you end up reading are in the same console.

Think of it as

Think of instrumentation as the plug and the backend as the socket. X-Ray gives you a matched pair — one plug, one socket, no adapters. OpenTelemetry is the standard plug that fits many sockets, which costs a little more setup and buys the ability to change socket later without rewiring the building.

text

# OpenTelemetry: instrument once, point the exporter at AWS
OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_ENDPOINT=<the AWS OTLP endpoint>
OTEL_TRACES_EXPORTER=otlp

Running both instrumentations at once without deciding which one owns propagation

Wrong

text

# X-Ray SDK on service A, OpenTelemetry on service B, no agreement on
# which trace context header is authoritative

Better

text

# Pick one propagation format for the whole request path, and configure
# the other side to read and emit it

What you see: Traces split at the boundary between the two services: each half is internally correct, and the service map shows two roots for what was one request.

Why: A trace only exists because every hop agrees on how the context is carried. Two instrumentation stacks that write different headers each start a fresh trace at the boundary, so the failure looks like a propagation bug rather than a configuration choice nobody made.

Choosing an instrumentation path

Choosing an instrumentation path
Situation	Reach for	Why
All-AWS stack, tracing needed today	X-Ray SDK	Header propagation and AWS SDK instrumentation come built in
Hybrid or multi-cloud, or a second observability vendor in play	OpenTelemetry (OTLP)	One instrumentation, several possible backends
Existing OpenTelemetry instrumentation from another platform	OpenTelemetry (OTLP)	Keep the instrumentation, change only the exporter
You want Prometheus-style metrics and PromQL alarms	OpenTelemetry metrics over OTLP	CloudWatch queries these with PromQL in Query Studio

Together

text

# The two data models differ where identity is concerned
CloudWatch metric   : namespace + name + up to 30 dimensions
OpenTelemetry metric: name + up to 150 labels, queried with PromQL

Remember: X-Ray SDK = AWS-native, least setup on an all-AWS stack. OpenTelemetry = an open standard AWS accepts over OTLP, with CloudWatch Application Signals on the other side, and portability as the reason to pay its setup cost. Whichever you pick, one context format must own the whole request path.

Attributing Latency Across Tiers

coreadvanced

A slow endpoint is not one number, it is a sum. Tracing splits that sum into named parts: time your own code spent computing, time waiting on a database, time waiting on another service, and time the request spent in transit. Each part has a different fix, so attributing the latency correctly is what stops you optimising the wrong thing.

Think of it as

A subsegment measures the caller's wait; the downstream service's own segment measures its work. The gap between the two is everything that is not work — connection setup, queueing, the network. That gap is invisible unless both ends are instrumented, which is why the comparison is the whole technique.

What we're doing: Decide whether a 4.2 second checkout needs a bigger database, a cache, or a code change.

attribution.txttext

Trace: POST /checkout, 4,200 ms total.

  auth-service subsegment      180 ms
  auth-service own segment     120 ms   -> 60 ms is not work

  pricing-service subsegment    90 ms   (parallel with auth? no — serial)

  SQL "SELECT ... JOIN ..."  3,900 ms   -> 93% of the request

  unaccounted                  120 ms   -> our own handler code

3: The 60 ms gap between the caller's view and the callee's view is connection setup and network. Real, but not the problem here.
6: auth and pricing do not depend on each other. Running them concurrently would remove 90 ms — worth doing, and still not the incident.
9: 3,900 ms in one query. Scaling the database instance up moves this number a little; fixing the query moves it by an order of magnitude.

Why this works: Without attribution, "checkout is slow" invites the most available fix — a bigger instance, or a cache in front. The trace says 93% of the time is one query, which makes a bigger instance an expensive way to not fix it and a cache a way to hide a query that will still be wrong on the first miss.

Adding a cache to hide a slow query instead of fixing it

Wrong

text

# Wrap the 3.9s query in a 5-minute cache and call it fixed

Better

text

# Fix the query (index, join shape, pagination) first; cache afterwards
# if the corrected query is still hot enough to be worth caching

What you see: p50 improves immediately and p99 does not move at all, because every cache miss still pays the full 3.9 seconds — and every deploy that flushes the cache turns into a self-inflicted incident.

Why: A cache changes how often you pay a cost, not the cost itself. With a 3.9-second miss, the tail latency stays exactly as bad and now also depends on cache warmth, which makes the system fragile in a new way at the moment it appears fixed.

Reading a 4.2s trace

Total segment: 4,200 ms

What the client experienced

Subsegment: auth service 180 ms

Its own segment says 120 ms — 60 ms is network + queueing

Subsegment: SQL query 3,900 ms

One query, no concurrency — this is the incident

Unaccounted: 120 ms

Inside the segment, outside every subsegment — your own code

Where the time went, and what to do about it

Where the time went, and what to do about it
Signature in the trace	What it means	The fix that helps
One long database subsegment	A slow query or a missing index	Fix the query; caching only hides it
Many short database subsegments in series	N+1 queries	Batch or join — more database capacity will not help
Round trip much larger than the downstream segment	Network, queueing, or connection acquisition	Connection pooling, keep-alive, co-location
Serial subsegments with no dependency between them	Missed concurrency	Issue the calls in parallel
Downstream throttles then retries	A quota, not slowness	Back off, or raise the quota
Large unaccounted time inside the segment	Your own CPU or blocking work	Profile the handler; tracing has taken you as far as it can

Together

text

# The N+1 signature — 60 subsegments of 40ms each, in series
segment  GET /orders                       2,480 ms
  subsegment  SELECT * FROM orders             35 ms
  subsegment  SELECT * FROM items WHERE id=1   40 ms
  subsegment  SELECT * FROM items WHERE id=2   41 ms
  ... 58 more ...

Remember: Split the total into: your own code (inside the segment, outside every subsegment), database time, downstream time, and the gap between the caller's round trip and the callee's own duration — which is network and queueing. Each part has a different fix.