beginnerintermediateadvanced

System Design · Section 61

Observability

Level: intermediate
Read: 16 min
Concepts: 3

Observability rests on three complementary pillars — logs (rich detail on one event), metrics (cheap, continuously-watchable aggregate trends) and traces (the path one request took across services) — none of which substitutes for the others. Making a system actually debuggable means consistently attaching a specific set of signals everywhere: request IDs to correlate within one hop, trace IDs to correlate across every hop, the service name on every signal, latency and error counts as basic per-service health, saturation as the earliest warning a resource is running out of room, and dependency metrics to tell whether a problem is a service's own or something it calls. Distributed tracing then has to work across two genuinely different propagation mechanisms: synchronous cross-service calls carry trace context on request headers riding the call itself, while asynchronous workflows have no live call to ride on and must have that context explicitly serialized into the message and extracted by the consumer, or the async leg becomes an invisible, disconnected trace.

This section

Observability Crash Course: Logs, Metrics, Traces ExplainedSystem Design Lab

What is true here

Logs, metrics and traces are complementary — metrics detect, traces localize, logs explain; none substitutes for the others.
Request IDs correlate within one hop; trace IDs correlate the same logical request across every hop — both only work if propagated, not regenerated per service.
Saturation is often the earliest warning sign of trouble, rising before error counts or latency visibly degrade.
Dependency metrics, recorded on the calling service's own side, are what separate "I am slow" from "something I depend on is slow."
Synchronous calls propagate trace context via headers; asynchronous workflows must serialize that context into the message itself, or the workflow silently loses tracing.

What you will be able to do

Explain what each of the three observability pillars is for and why none of them can replace the other two
Name the concrete signals a request needs (IDs, service name, latency, errors, saturation, dependency metrics) and why each earns its place
Propagate trace context correctly across both a synchronous service call and an asynchronous, queue-based workflow
Recognize saturation as a leading indicator worth alerting on before errors appear

The three pillars

Logs, metrics and traces as complementary data shapes, each answering a question the others cannot.

Three pillars: logs, metrics and traces

coreintermediate

Logs, metrics and traces are three different shapes of data about the same running system, and each answers a question the other two cannot answer well. A log is a discrete, timestamped record of one event — rich in detail about exactly what happened, but expensive to store and slow to query across a whole fleet at scale. A metric is a numeric measurement aggregated over time (a count, a rate, a gauge, a histogram bucket) — cheap to store and fast to query, but it tells you something is wrong without telling you which request or which user was affected. A trace follows one request as it moves across multiple services, recording how long each hop took and how the hops relate to each other — it answers "where did this specific request spend its time" in a way neither logs nor metrics alone can. None of the three replaces the other two; a mature observability setup carries all three and correlates them together.

Think of it as

Think of a hospital: metrics are the vital-signs monitor at the nurses' station showing heart rate and blood pressure trends for every patient at a glance — great for spotting that something is wrong right now, useless for explaining why. Logs are the detailed nurse's notes on one patient's chart — rich and specific, but you would not want to read every patient's full chart just to find who is in trouble. A trace is the patient's full visit itinerary — check-in, triage, X-ray, lab work, doctor consult — showing how long each stop took and in what order, which is exactly what you need when one patient's visit took six hours and you want to know which single stop caused the delay.

text

// The same incident, seen through each pillar

metric:  http_request_duration_seconds{route="/checkout"} p99 spikes to 4.2s
log:     2026-08-25T10:03:11Z ERROR checkout-svc req_id=abc123 "payment gateway timeout after 4000ms"
trace:   req_id=abc123
           checkout-svc     [====================] 4210ms
             payment-svc    [==================]   4180ms  <- almost all the time
               db-query     [==]                     220ms

What we're doing: Use all three pillars together to diagnose one slow checkout request.

investigation.txttext

1. Dashboard alert fires: checkout p99 latency is
   4.2s, up from a normal 300ms.
2. Metrics alone cannot say which requests are slow
   or why -- only that the aggregate distribution
   shifted.
3. Filter traces for checkout-svc with duration >
   2s; find one recent example, req_id=abc123.
4. The trace shows checkout-svc's call into
   payment-svc took 4180ms of the 4210ms total --
   the time is almost entirely in one downstream
   span, not in checkout-svc itself.
5. Pull logs for payment-svc filtered to
   req_id=abc123; find "payment gateway timeout
   after 4000ms".
6. Root cause: an upstream payment gateway is timing
   out, not a bug in either of our own services.

3: The metric told us something was wrong; it could not tell us which request or which service to look at next.
8: The trace narrows the problem from "checkout is slow" to "one specific downstream span is slow" without reading a single log line.
13: Only the log line has the actual error text — the trace shows where the time went, not why.

Why this works: This is the intended division of labor: a metric detects the anomaly at a glance, a trace localizes it to a specific service and span, and a log explains the specific cause at that span — skipping any one of the three would have left a real gap in this diagnosis.

Trying to diagnose a specific request using only dashboards

Wrong

text

# On-call sees the p99 latency dashboard spike
# and starts manually re-running the checkout flow
# hoping to reproduce it, with no request ID at hand

Better

text

# On-call pulls a sample of trace IDs for requests
# that fell inside the spike window and duration
# threshold from the tracing backend, then jumps
# straight to those traces and their correlated logs

What you see: The on-call engineer spends 30+ minutes trying to manually reproduce a transient issue that already happened and is sitting recorded in the tracing and logging backends, because the dashboard alone gives no way to identify which specific request or dependency was actually responsible.

Why: A metrics dashboard has no concept of an individual request — it is already an aggregate. Without pivoting to traces and logs for actual affected request IDs, the engineer is reduced to guessing and reproducing blind, when the real evidence already exists and only needs to be looked up.

How the three pillars connect during one investigation

Metric dashboard

p99 latency spikes — something is wrong

Trace for a slow request

which service and span ate the time

Logs for that span

exact error message and context

What each pillar is actually good and bad at

What each pillar is actually good and bad at
Pillar	Best for	Weak for
Logs	Exact detail of one event, ad hoc investigation	Aggregating patterns across millions of events cheaply
Metrics	Trends, dashboards, alerting thresholds, cheap high-cardinality-free queries	Explaining why one specific request failed
Traces	Seeing where time went across services for one request	Answering "how often does this happen system-wide"

Remember: Metrics detect that something is wrong system-wide and are cheap to watch continuously; traces localize the problem to a specific service and span for one request; logs explain the specific cause once localized. The three are complementary — a request/trace ID is the thread that lets you pivot from one pillar to the next during a real investigation.

Signals and tracing in practice

The concrete signal checklist for every request, and how a single trace stays connected across both synchronous and asynchronous hops.

Core signals: IDs, service names, latency, errors, saturation and dependencies

coreintermediate

Turning raw logs and metrics into something actually useful for debugging means consistently attaching a specific set of signals to every request, everywhere. A request ID uniquely identifies one request as it is handled within a single service (or one hop), while a trace ID identifies that same logical request as it crosses multiple services — both must be generated (or accepted from an inbound header) at the edge and propagated on every downstream call, or the correlation breaks. The service name on every log line and metric is what lets you tell which of possibly hundreds of services actually emitted a given signal. Latency and error counts are the two most basic health signals for any single service. Saturation (how full a resource is — CPU, memory, connection pool, queue depth) is the leading indicator that a service is about to start failing, often before error counts rise at all. Dependency metrics — the latency and error rate of the services or datastores a given service calls — are what let you tell whether a service's own problems are self-inflicted or caused by something it depends on.

Think of it as

Think of these signals like the standard information printed on every shipping label and package a courier company handles: a tracking number (request/trace ID) so you can look up any one package's specific journey, the depot name stamped on it (service name) so you know which facility handled it, how long it sat at each stop (latency), how many packages got damaged at each stop (error counts), how full each depot's truck bay is right now (saturation, the leading warning sign a depot is about to fall behind), and which downstream carrier each depot handed packages off to and how that carrier performed (dependency metrics). Strip any one of these off the label and a package that goes missing becomes far harder to trace.

text

// Structured log line carrying every signal at once
{
  "ts": "2026-08-25T10:03:11Z",
  "service": "checkout-svc",
  "trace_id": "abc123",
  "request_id": "req-9f2e",
  "latency_ms": 4210,
  "status": "error",
  "dependency": {
    "name": "payment-svc",
    "latency_ms": 4180,
    "status": "timeout"
  }
}
// Saturation is usually a separate metric, not a log line:
pool_connections_in_use / pool_connections_max  // e.g. 98/100 -> near saturation

What we're doing: Instrument a request end to end so every signal is present when something goes wrong.

signal-checklist.txttext

1. Gateway receives request with no inbound trace
   header -- it mints trace_id=abc123 and a fresh
   request_id, attaches both to every downstream call.
2. checkout-svc receives the call, reads trace_id
   from the header (does NOT generate a new one),
   generates its own request_id for its local logs.
3. checkout-svc records: service=checkout-svc,
   latency, error count, and current connection-pool
   saturation.
4. checkout-svc calls payment-svc; it records this
   call as a dependency metric (name, latency,
   status) on its own side.
5. payment-svc times out; checkout-svc's dependency
   metric shows the timeout, its own latency metric
   shows 4210ms, and its error count increments --
   all tagged with the same trace_id=abc123.

2: Minting the trace ID at the edge and forwarding it (not regenerating per-service) is what keeps every downstream signal correlatable to the same logical request.
10: Saturation is captured even though this request did not fail due to resource exhaustion -- it needs to be always-on, not added reactively after an incident.
13: The dependency metric is recorded on checkout-svc's own side about payment-svc, which is what lets checkout-svc's own dashboards distinguish "I am slow" from "payment-svc is slow".

Why this works: Every signal in the list is present by the time the timeout happens, which is precisely what makes it possible to tell, without guessing, that payment-svc's timeout — not checkout-svc's own code — caused the failure.

Generating a fresh trace ID at each service instead of propagating the inbound one

Wrong

text

// Every service does this on request entry
def handle_request(req):
    trace_id = generate_new_id()  # ignores any
                                   # inbound header
    log(trace_id, ...)

Better

text

def handle_request(req):
    trace_id = req.headers.get('trace-id') \
               or generate_new_id()  # only mint if
                                       # truly absent
    forward_headers['trace-id'] = trace_id
    log(trace_id, ...)

What you see: Every service's logs for what was actually one user request show a different, unrelated trace ID, so there is no single query that pulls the full cross-service picture — engineers instead have to correlate by timestamp and guesswork, which fails as soon as traffic volume is more than trivial.

Why: A trace ID is only useful as a correlation key if every hop uses the same one. Generating a new ID instead of forwarding the inbound one silently defeats the entire purpose of having one, even though each individual service still appears to be "using trace IDs" in isolation.

How one trace ID threads through service names and their dependency calls

API gateway

mints trace_id=abc123

checkout-svc

latency 4210ms, error

payment-svc

dependency: 4180ms, timeout

The seven signals and what each one is for

The seven signals and what each one is for
Signal	Scope	What it answers
Request ID	One request, one service/hop	"Which log lines belong to this exact request here?"
Trace ID	One request, across all services	"Which spans/logs across the whole system belong to this request?"
Service name	Every signal emitted	"Which service produced this signal?"
Latency	Per service (or per endpoint)	"How long is this service taking to respond?"
Error counts	Per service (or per endpoint)	"How often is this service failing?"
Saturation	Per resource (CPU, memory, pool, queue)	"How close is this service to running out of capacity?"
Dependency metrics	Per downstream call	"Is this service's problem its own, or a dependency's?"

Remember: Request IDs correlate within one hop, trace IDs correlate across every hop — both only work if propagated on every downstream call rather than regenerated per service. Latency and error counts describe a service itself; saturation is the earliest warning sign of trouble, often rising well before either does; dependency metrics tell you whether a problem is a service's own or something it calls.

Tracing cross-service requests and asynchronous workflows

coreintermediate

A distributed trace represents one request as a tree of spans, where each span is one unit of work (an HTTP call, a database query, a function) with a start time, a duration, and a parent-child relationship to the span that caused it. For synchronous cross-service requests, the trace context (trace ID plus the current span ID, so the callee knows whose child it is) rides along as HTTP headers on the call itself — this is what lets a single trace stitch together spans from many different services into one coherent tree. Asynchronous workflows break that mechanism: when a message is published to a queue and processed later by a consumer, there is no in-flight HTTP call to carry headers on, so the trace context has to be explicitly serialized into the message itself (as message attributes or a header field within the message payload) and read back out by the consumer to continue the same trace rather than start a disconnected new one.

Think of it as

A synchronous cross-service trace is like a relay race where the baton (trace context) is physically handed from one runner to the next mid-stride — as long as every runner grabs the baton before running, the whole race is obviously one continuous event. An asynchronous workflow is more like mailing a letter that will be opened and acted on days later by someone in a different building: there is no hand-to-hand handoff, so if you want the eventual reply to be recognized as part of the same conversation, you have to write a reference number inside the letter itself — if you forget, the person who opens it later has no way of knowing which conversation it belongs to, and it looks like a conversation that started out of nowhere.

text

// Synchronous: trace context rides HTTP headers
GET /charge HTTP/1.1
traceparent: 00-abc123...-span456...-01

// Asynchronous: trace context is carried inside
// the message itself, since there is no live call
{
  "event": "order.placed",
  "trace_context": {
    "trace_id": "abc123...",
    "span_id": "span789..."
  },
  "payload": { "order_id": 42 }
}
// consumer extracts trace_context and starts its
// span as a CHILD of span789, not a new root

What we're doing: Follow one order through a synchronous API call and an asynchronous fulfillment step as a single trace.

async-trace.txttext

1. Client calls order-api; it has no inbound trace
   context so it mints trace_id=abc123, starts a
   root span "place-order".
2. order-api calls inventory-svc synchronously via
   HTTP; trace_id and parent span ID ride on the
   traceparent header. inventory-svc's span is a
   child of "place-order".
3. order-api publishes an "order.placed" event to a
   queue, serializing {trace_id, span_id} into the
   message payload before publishing.
4. order-api returns 202 Accepted to the client --
   the synchronous part of the trace ends here.
5. Minutes later, fulfillment-worker consumes the
   message, extracts trace_context from the payload,
   and starts its own span as a child of the
   original "place-order" span.
6. The tracing backend shows one trace containing
   all three spans, even though step 5 happened
   minutes after step 4 returned.

8: This is the moment synchronous propagation (headers) is no longer available -- the response has already gone back to the client.
11: Extracting trace_context here, rather than ignoring it, is what keeps the asynchronous step attached to the original request instead of looking like an unrelated event.
14: The trace correctly spans minutes and a queue hop, proving that trace duration is about logical request scope, not wall-clock proximity or a single live connection.

Why this works: The example deliberately includes both propagation styles in one trace to show that the mechanism differs (headers vs. serialized payload) but the goal is identical: every span, however it was reached, ends up attached to the same trace ID as a child of the span that caused it.

Assuming a queue consumer automatically inherits the producer's trace

Wrong

text

// Producer publishes with no trace context at all
queue.publish({ event: "order.placed", order_id: 42 })

// Consumer starts a brand-new trace on every message
def on_message(msg):
    with tracer.start_span("process-order"):  # new root
        ...

Better

text

// Producer serializes the current trace context
queue.publish({
  event: "order.placed", order_id: 42,
  trace_context: tracer.current_context(),
})

def on_message(msg):
    ctx = extract(msg["trace_context"])
    with tracer.start_span("process-order", parent=ctx):
        ...

What you see: The tracing backend shows the fulfillment work as a completely separate, unrelated trace with no connection back to the order-placement request that triggered it, so a slow or failing fulfillment step cannot be traced back to which original request caused it — engineers have to correlate by order ID in logs by hand instead of following one trace.

Why: Trace context does not propagate automatically across a queue the way it does across a direct HTTP call — there is no shared connection or header mechanism to ride on. Unless the producer explicitly serializes it into the message and the consumer explicitly extracts it, the two ends of the same logical workflow have no structural link, and the tracing system has no way to know they belong together.

One trace spanning a synchronous call and an asynchronous handoff

order-api

root span; trace_id=abc123

message queue

trace_context serialized into message

fulfillment-worker

extracts trace_context; child span of order-api

How trace context travels, by call style

How trace context travels, by call style
Call style	How context travels	Common failure mode
Synchronous HTTP/RPC	Headers on the request itself (e.g. traceparent)	A hop that strips unknown headers silently breaks the chain
Message queue / event	Serialized into message attributes or payload	Consumer ignores the field and starts a fresh, disconnected trace
Scheduled/batch job	Often none — job runs independently of any request	Batch work is invisible in tracing entirely unless deliberately given its own root span

Remember: Synchronous cross-service calls propagate trace context via request headers; asynchronous workflows have no live call to ride on, so the context must be explicitly serialized into the message and extracted by the consumer as a child span, not a fresh trace. A trace spans wall-clock time and queue hops just as validly as it spans a single synchronous call chain — skipping context propagation on any hop, sync or async, silently breaks that one trace into disconnected pieces.