Observability

Logs, metrics and traces are three different shapes of data about the same running system, and each answers a question the other two cannot answer well. A log is a discrete, timestamped record of one event — rich in detail about exactly what happened, but expensive to store and slow to query across a whole fleet at scale. A metric is a numeric measurement aggregated over time (a count, a rate, a gauge, a histogram bucket) — cheap to store and fast to query, but it tells you something is wrong without telling you which request or which user was affected. A trace follows one request as it moves across multiple services, recording how long each hop took and how the hops relate to each other — it answers "where did this specific request spend its time" in a way neither logs nor metrics alone can. None of the three replaces the other two; a mature observability setup carries all three and correlates them together.

// The same incident, seen through each pillar

metric:  http_request_duration_seconds{route="/checkout"} p99 spikes to 4.2s
log:     2026-08-25T10:03:11Z ERROR checkout-svc req_id=abc123 "payment gateway timeout after 4000ms"
trace:   req_id=abc123
           checkout-svc     [====================] 4210ms
             payment-svc    [==================]   4180ms  <- almost all the time
               db-query     [==]                     220ms

Turning raw logs and metrics into something actually useful for debugging means consistently attaching a specific set of signals to every request, everywhere. A request ID uniquely identifies one request as it is handled within a single service (or one hop), while a trace ID identifies that same logical request as it crosses multiple services — both must be generated (or accepted from an inbound header) at the edge and propagated on every downstream call, or the correlation breaks. The service name on every log line and metric is what lets you tell which of possibly hundreds of services actually emitted a given signal. Latency and error counts are the two most basic health signals for any single service. Saturation (how full a resource is — CPU, memory, connection pool, queue depth) is the leading indicator that a service is about to start failing, often before error counts rise at all. Dependency metrics — the latency and error rate of the services or datastores a given service calls — are what let you tell whether a service's own problems are self-inflicted or caused by something it depends on.

// Structured log line carrying every signal at once
{
  "ts": "2026-08-25T10:03:11Z",
  "service": "checkout-svc",
  "trace_id": "abc123",
  "request_id": "req-9f2e",
  "latency_ms": 4210,
  "status": "error",
  "dependency": {
    "name": "payment-svc",
    "latency_ms": 4180,
    "status": "timeout"
  }
}
// Saturation is usually a separate metric, not a log line:
pool_connections_in_use / pool_connections_max  // e.g. 98/100 -> near saturation

A distributed trace represents one request as a tree of spans, where each span is one unit of work (an HTTP call, a database query, a function) with a start time, a duration, and a parent-child relationship to the span that caused it. For synchronous cross-service requests, the trace context (trace ID plus the current span ID, so the callee knows whose child it is) rides along as HTTP headers on the call itself — this is what lets a single trace stitch together spans from many different services into one coherent tree. Asynchronous workflows break that mechanism: when a message is published to a queue and processed later by a consumer, there is no in-flight HTTP call to carry headers on, so the trace context has to be explicitly serialized into the message itself (as message attributes or a header field within the message payload) and read back out by the consumer to continue the same trace rather than start a disconnected new one.

// Synchronous: trace context rides HTTP headers
GET /charge HTTP/1.1
traceparent: 00-abc123...-span456...-01

// Asynchronous: trace context is carried inside
// the message itself, since there is no live call
{
  "event": "order.placed",
  "trace_context": {
    "trace_id": "abc123...",
    "span_id": "span789..."
  },
  "payload": { "order_id": 42 }
}
// consumer extracts trace_context and starts its
// span as a CHILD of span789, not a new root