beginnerintermediateadvanced

System Design · Monitoring and Alerting

Monitoring and Alerting

Alert on what users feel, not on infrastructure noise, and know the operational checks that decide restart vs rotation

Six symptomserrors, tail latency, saturation, queue age, freshness, availability
Not infra noiseCPU/disk alone maps to no fixed user impact
Liveness vs readinessrestart vs pull-from-rotation
Synthetic monitoringscheduled, external, real-user-like probes

1.Symptom-based alerting: the six signals tied to user impact

A symptom is something a user (or a downstream system acting on their behalf) actually experiences — a failed request, a slow page, a stale dashboard. An alert should fire on symptoms, not on whatever happens to be easy to measure inside a server. The roadmap names six symptom signals that cover most systems: error rate (the fraction of requests failing), tail latency (p95/p99, not the average, since the average hides exactly the slow requests users notice), saturation (how close a resource is to its limit — queue depth, connection pool usage, thread pool occupancy), queue age (how long the oldest unprocessed item has been waiting, which is what a user or downstream consumer actually feels), data freshness (how stale a cache, replica, or materialized view is relative to the source of truth), and availability (whether the service can be reached and used at all, as an end-to-end measurement). Each of these maps to something a real person or dependent service notices going wrong — which is the entire design principle.

Key features

Error rate: fraction of requests failing — usually alerted as a ratio (errors/total) over a rolling window, not a raw count, so it scales with traffic
Tail latency (p95/p99): the average can look healthy while a meaningful fraction of users wait far longer than that — alert on the percentile, not the mean
Saturation: how full a resource is relative to its limit (queue depth, connection pool, thread pool, disk) — a leading indicator that precedes errors and latency
Queue age: the wait time of the oldest unprocessed item — a growing queue with fast individual processing can still mean every user is waiting minutes
Data freshness: staleness of a cache, read replica, or materialized view versus the source of truth — silently stale data causes wrong answers with no errors at all
Availability: whether the service can be reached and used end-to-end — usually measured as a black-box, external probe rather than trusting the service's own report of itself

The six symptom signals and what a user actually feels when each fires

The six symptom signals and what a user actually feels when each fires
Signal	What it measures	What the user feels
Error rate	Fraction of requests returning a failure	Action fails outright — a submit button that errors
Tail latency (p95/p99)	Response time at the slow percentile, not the average	Page or API call visibly hangs, even though "most" requests are fine
Saturation	How close a resource is to its capacity limit	Slowness and errors that are about to start, or just started
Queue age	Wait time of the oldest unprocessed item	A background job, notification, or order confirmation arrives late
Data freshness	Staleness of derived data versus the source of truth	Wrong or outdated information shown with no visible error
Availability	Whether the service is reachable and usable at all	Total outage — nothing loads

Six symptom signals, one shared property: user-felt impact

Error rate

requests failing

Tail latency

p95/p99, not mean

Saturation

resource near limit

Queue age

oldest item waits

Data freshness

staleness vs source

Availability

reachable end-to-end

Command

yaml

# Symptom-based alert definitions (conceptual)
- name: checkout-error-rate
  expr: rate(errors[5m]) / rate(requests[5m]) > 0.02
  for: 5m
  severity: page

- name: checkout-p99-latency
  expr: histogram_quantile(0.99, latency_bucket) > 800ms
  for: 10m
  severity: page

- name: order-queue-age
  expr: oldest_unprocessed_item_age_seconds > 120
  for: 5m
  severity: page

Alert on what a user or dependent system actually experiences — error rate, tail latency (p95/p99, never the average), saturation, queue age, data freshness, and availability — not on whichever internal metric happens to be easy to scrape. A symptom alert tells the on-call engineer that something is actually wrong for someone; everything else is diagnostic detail that belongs in a runbook link, not a separate page.

2.Avoiding infrastructure-noise-only alerting

Infrastructure noise is a raw resource metric — CPU percent, disk percent, memory percent, network throughput — alerted on directly, with no connection to whether users are actually affected. It is tempting because these metrics are always available, cheap to collect, and easy to threshold ("page me if CPU > 80%"). The problem is that most infrastructure metrics have no fixed, universal relationship to user impact: a server can run at 95% CPU all day serving every request within its latency target (because it was provisioned to run hot), and a server can sit at 20% CPU while genuinely failing users (because the bottleneck is a lock, a slow downstream dependency, or connection exhaustion that CPU never reflects). Alerting only on infrastructure noise produces two failure modes at once: constant false pages for resource levels that are actually fine, and missed real incidents where the user-facing symptom fires but the infrastructure metric never crosses its threshold. The fix is not to delete infrastructure metrics — they stay valuable as saturation signals and diagnostic context — but to stop treating them as the primary page-worthy alert.

Key features

Infrastructure metrics (CPU, disk, memory, network throughput) have no fixed threshold that maps to user impact — the same 90% CPU reading can be fine on one service and a crisis on another
Alerting only on infrastructure noise produces two failures at once: false pages when the resource is fine, and missed incidents when the resource never crosses threshold despite real user impact
A high resource reading is only actionable through a causal chain to a symptom — it is a hypothesis about a cause, not itself a page-worthy event
Infrastructure metrics remain useful as saturation signals (see symptom-based alerting) and as diagnostic context attached to a symptom alert's runbook
A common middle ground: alert directly on symptoms, but keep infrastructure dashboards close at hand so the responder's first diagnostic step is fast, not the alert itself

Infrastructure-noise alerting versus symptom-based alerting for the same incident

Infrastructure-noise alerting versus symptom-based alerting for the same incident
Property	Infrastructure-noise alert	Symptom-based alert
What it measures	A raw resource level (CPU%, disk%, memory%)	What the user or dependent system actually experiences
False-positive risk	High — many high-resource states are perfectly healthy	Low — the alert only fires when something is actually broken
False-negative risk	High — real incidents can occur with resources nowhere near threshold	Low, if the symptom set is complete (error rate, latency, saturation, queue age, freshness, availability)
Best use	Diagnostic context once a symptom alert has fired	The primary trigger that pages someone

Where a raw resource metric belongs versus where an alert should trigger from

Infrastructure-noise alerting

Pages on CPU/disk/memory crossing a fixed threshold
No universal threshold maps to user impact
Pages during normal, healthy high-load periods
Misses incidents where the bottleneck is not resource-shaped

Symptom-based alerting

Pages on error rate, tail latency, queue age, availability
Directly tied to what a user or dependent system feels
Infrastructure metrics stay as linked diagnostic dashboards
Catches non-resource-shaped incidents (locks, slow dependencies)

Command

yaml

# Infrastructure noise treated as a diagnostic signal,
# not a page trigger
dashboards:
  - checkout-service-resources   # CPU, memory, disk, network
    linked_from: checkout-error-rate-alert   # the actual page

# The alert that pages is the symptom, with the
# resource dashboard one click away in the runbook

A raw resource reading (CPU, disk, memory) has no fixed threshold that maps to user impact, so it makes a poor primary alert: it pages during healthy high-load periods and stays silent through incidents that are not resource-shaped (locks, slow dependencies, connection exhaustion). Keep infrastructure metrics as saturation signals and runbook-linked diagnostic dashboards; let the page itself come from a symptom.

3.Health checks: readiness vs liveness, and synthetic monitoring

A liveness check answers one narrow question: "is this process still running and not permanently stuck?" If it fails, the correct response is to restart the process — it is not coming back on its own. A readiness check answers a different question: "is this process currently able to serve traffic right now?" A process can be alive (not stuck, not crashed) while not ready — still loading a cache on startup, temporarily disconnected from its database, or deliberately draining connections before a shutdown. Restarting on a readiness failure would be wrong (the process is fine, it just needs a moment or a dependency to recover); routing traffic to it would also be wrong (it cannot serve requests correctly yet). Orchestrators like Kubernetes act on these differently: a failed liveness probe triggers a restart, a failed readiness probe removes the instance from the load balancer's rotation without restarting it. Synthetic monitoring is a third, complementary check: rather than asking one instance about itself, an external agent periodically performs a real user-like action (log in, load a page, complete a checkout) from outside the system, on a schedule, whether or not any real user happens to be doing so at that moment — catching failures that internal health checks cannot see, such as a broken DNS record, an expired TLS certificate, or a working backend behind a broken load balancer rule.

Key features

Liveness check answers "is the process stuck or crashed?" — failure triggers a restart, since a stuck process will not self-recover
Readiness check answers "can this instance serve traffic right now?" — failure removes it from load-balancer rotation without restarting it
A process can be alive but not ready (warming a cache, draining before shutdown, temporarily lost a dependency) — conflating the two causes wrong responses in both directions
Synthetic monitoring runs real user-like actions from outside the system on a fixed schedule, independent of real traffic — it catches failures internal health checks cannot see (DNS, TLS expiry, load-balancer misrouting, third-party dependency outages)
Kubernetes formalizes this distinction directly: livenessProbe controls container restarts, readinessProbe controls Service/Endpoint membership
Synthetic checks complement, not replace, real-traffic symptom monitoring — a synthetic probe only samples the specific paths it was scripted to test, at the frequency it runs

The three operational checks and what triggers on each

The three operational checks and what triggers on each
Check	Question it answers	What happens on failure
Liveness	Is the process stuck or crashed?	Orchestrator restarts the process/container
Readiness	Can this instance serve traffic right now?	Instance removed from load-balancer rotation, not restarted
Synthetic monitoring	Can a real user complete a real action, end-to-end, right now?	Alert fires — catches DNS, TLS, and routing failures no internal check sees

A pod's health across startup, steady state, and a dependency blip

readinessprobe passesdependency drops,readiness failsdependencyrecoverslivenessprobe fails

Starting

start

Alive + Ready

Alive, not Ready

Restarted

end

Command

yaml

# Kubernetes probe configuration (conceptual)
livenessProbe:
  httpGet:
    path: /healthz        # "am I stuck?" -- cheap, no dependency checks
    port: 8080
  periodSeconds: 10
  failureThreshold: 3       # restart after 3 consecutive failures

readinessProbe:
  httpGet:
    path: /readyz          # "can I serve traffic?" -- checks real deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 2       # pulled from rotation after 2 failures

Liveness asks "is the process stuck?" and its failure means restart; readiness asks "can this instance serve traffic right now?" and its failure means pull-from-rotation without restarting. Never bundle dependency checks into the liveness probe, and keep the liveness check itself cheap. Synthetic monitoring is a third, complementary layer — a scheduled, external, real-user-like check that catches DNS, TLS, and routing failures no internal probe can see.

Spotted a mistake on this page? Tell us.

Read Sheet