beginnerintermediateadvanced

System Design · Section 62

Monitoring and Alerting

Level: intermediate
Read: 18 min
Concepts: 3

A good alert fires on what a user or dependent system actually experiences — error rate, tail latency (p95/p99, not the average), saturation, queue age, data freshness, and availability — rather than on whatever internal metric happens to be cheapest to scrape. Alerting only on infrastructure noise (raw CPU, disk, or memory percentages) fails in both directions at once: it pages during perfectly healthy high-load periods, and it stays silent through real incidents that are not resource-shaped, such as thread starvation on a slow downstream dependency. Health checks complete the operational picture at the individual-instance level: a liveness check asks whether a process is stuck (failure means restart), a readiness check asks whether an instance can serve traffic right now (failure means removal from load-balancer rotation, not a restart), and synthetic monitoring adds a third, complementary layer — a scheduled, external probe that performs a real user-like action regardless of real traffic, catching failures like a broken DNS record or expired certificate that no internal check or symptom metric would ever see.

This section

Design Metrics Monitoring & Alerting System: System Design Interview (Stripe & Amazon Offers)TechPrep

What is true here

Alert on the six symptom signals tied to user impact — error rate, tail latency, saturation, queue age, data freshness, availability — not on internal metrics with no fixed relationship to impact.
Infrastructure-noise-only alerting (CPU/disk/memory) both false-pages during healthy load and misses real incidents that are not resource-shaped.
Liveness failure triggers a restart; readiness failure triggers removal from traffic rotation without a restart — conflating the two causes restart storms or traffic routed to instances that cannot serve it.
Synthetic monitoring runs scheduled, external, real-user-like checks that catch failures (DNS, TLS, routing) invisible to both internal health checks and real-traffic symptom metrics.

What you will be able to do

Design alerts around the six symptom signals rather than raw internal metrics
Recognize and avoid the infrastructure-noise-only anti-pattern, including its alert-fatigue and missed-incident failure modes
Correctly separate liveness from readiness so restarts and traffic routing each happen for the right reason
Add synthetic monitoring to catch failure classes neither health checks nor symptom metrics can see

What to alert on

The six user-impact symptom signals, and the infrastructure-noise anti-pattern to avoid instead.

Symptom-based alerting: the six signals tied to user impact

coreintermediate

A symptom is something a user (or a downstream system acting on their behalf) actually experiences — a failed request, a slow page, a stale dashboard. An alert should fire on symptoms, not on whatever happens to be easy to measure inside a server. The roadmap names six symptom signals that cover most systems: error rate (the fraction of requests failing), tail latency (p95/p99, not the average, since the average hides exactly the slow requests users notice), saturation (how close a resource is to its limit — queue depth, connection pool usage, thread pool occupancy), queue age (how long the oldest unprocessed item has been waiting, which is what a user or downstream consumer actually feels), data freshness (how stale a cache, replica, or materialized view is relative to the source of truth), and availability (whether the service can be reached and used at all, as an end-to-end measurement). Each of these maps to something a real person or dependent service notices going wrong — which is the entire design principle.

Think of it as

Think of a car dashboard built for the driver, not the mechanic. The driver needs to know "the engine is overheating" (a symptom that affects whether the car can be driven right now) — not "cylinder 3 coolant sensor reads 4.7 ohms" (a raw internal reading that may or may not mean anything is actually wrong). A mechanic cares about the sensor reading during diagnosis; the driver needs the dashboard to escalate only when something changes what they can actually do with the car. Symptom-based alerts are the dashboard warning lights; raw infrastructure metrics are the sensor readings a mechanic pulls up after the light is already on.

yaml

# Symptom-based alert definitions (conceptual)
- name: checkout-error-rate
  expr: rate(errors[5m]) / rate(requests[5m]) > 0.02
  for: 5m
  severity: page

- name: checkout-p99-latency
  expr: histogram_quantile(0.99, latency_bucket) > 800ms
  for: 10m
  severity: page

- name: order-queue-age
  expr: oldest_unprocessed_item_age_seconds > 120
  for: 5m
  severity: page

What we're doing: Turn a vague "checkout feels slow sometimes" complaint into symptom-based alert coverage.

checkout-slos.yamlyaml

slo: checkout-availability
  target: 99.9% success over 30d
  alert_on: error_rate > 0.1% sustained 5m

slo: checkout-latency
  target: p99 < 800ms over 30d
  alert_on: p99 > 800ms sustained 10m

slo: checkout-queue-health
  target: oldest pending order < 60s
  alert_on: queue_age > 120s sustained 5m

2: Error rate is a ratio over a rolling window, not a raw error count — the same absolute count of errors means something different at 100 req/s versus 10,000 req/s.
5: p99, not the average — a small but real fraction of checkouts hanging is exactly what an average latency metric hides.
8: Queue age catches a failure mode neither error rate nor latency sees: individual requests can be fast and successful while a backlog quietly grows behind them.

Why this works: The vague complaint "checkout feels slow" is actually three separate, independently-alertable symptoms — outright failures, individually slow requests, and a growing backlog — and each needs its own threshold because they can fail independently of one another.

Setting one alert threshold for "errors" that conflates hard failures with soft degradation

Wrong

yaml

- name: checkout-problems
  expr: error_count > 10
  for: 1m
  severity: page

Better

yaml

- name: checkout-error-rate
  expr: rate(errors[5m]) / rate(requests[5m]) > 0.01
  for: 5m
  severity: page

- name: checkout-p99-latency
  expr: histogram_quantile(0.99, latency_bucket) > 800ms
  for: 10m
  severity: ticket   # degraded but not down

What you see: A raw error count of "10" pages the team identically whether it happened during 50 requests (a 20% failure rate — real outage) or 500,000 requests (a 0.002% failure rate — background noise) — the on-call engineer cannot tell severity from the alert itself and has to go look every time.

Why: A count has no denominator, so it cannot express rate, and it conflates two different situations that need different responses: a small number of slow-but-successful requests is degradation worth a ticket, while a spike in outright failures is an outage worth a page. Splitting the threshold by symptom, and expressing error alerts as a rate rather than a count, lets severity be read directly off which alert fired.

Six symptom signals, one shared property: user-felt impact

Error rate

requests failing

Tail latency

p95/p99, not mean

Saturation

resource near limit

Queue age

oldest item waits

Data freshness

staleness vs source

Availability

reachable end-to-end

The six symptom signals and what a user actually feels when each fires

The six symptom signals and what a user actually feels when each fires
Signal	What it measures	What the user feels
Error rate	Fraction of requests returning a failure	Action fails outright — a submit button that errors
Tail latency (p95/p99)	Response time at the slow percentile, not the average	Page or API call visibly hangs, even though "most" requests are fine
Saturation	How close a resource is to its capacity limit	Slowness and errors that are about to start, or just started
Queue age	Wait time of the oldest unprocessed item	A background job, notification, or order confirmation arrives late
Data freshness	Staleness of derived data versus the source of truth	Wrong or outdated information shown with no visible error
Availability	Whether the service is reachable and usable at all	Total outage — nothing loads

Remember: Alert on what a user or dependent system actually experiences — error rate, tail latency (p95/p99, never the average), saturation, queue age, data freshness, and availability — not on whichever internal metric happens to be easy to scrape. A symptom alert tells the on-call engineer that something is actually wrong for someone; everything else is diagnostic detail that belongs in a runbook link, not a separate page.

Avoiding infrastructure-noise-only alerting

coreintermediate

Infrastructure noise is a raw resource metric — CPU percent, disk percent, memory percent, network throughput — alerted on directly, with no connection to whether users are actually affected. It is tempting because these metrics are always available, cheap to collect, and easy to threshold ("page me if CPU > 80%"). The problem is that most infrastructure metrics have no fixed, universal relationship to user impact: a server can run at 95% CPU all day serving every request within its latency target (because it was provisioned to run hot), and a server can sit at 20% CPU while genuinely failing users (because the bottleneck is a lock, a slow downstream dependency, or connection exhaustion that CPU never reflects). Alerting only on infrastructure noise produces two failure modes at once: constant false pages for resource levels that are actually fine, and missed real incidents where the user-facing symptom fires but the infrastructure metric never crosses its threshold. The fix is not to delete infrastructure metrics — they stay valuable as saturation signals and diagnostic context — but to stop treating them as the primary page-worthy alert.

Think of it as

Think of a restaurant kitchen during a dinner rush. The kitchen being "at 95% capacity" (every burner in use, every cook busy) is completely normal on a busy Friday night — it is not a problem in itself. What actually tells the manager whether to worry is a symptom: are tickets taking too long to reach the pass, are dishes coming back wrong, are customers walking out. A manager who pages the head chef every time the kitchen looks "at capacity" gets paged every Friday for nothing, and might miss the night the kitchen is at 40% capacity but the one working oven is broken and every dish is late anyway.

yaml

# Infrastructure noise treated as a diagnostic signal,
# not a page trigger
dashboards:
  - checkout-service-resources   # CPU, memory, disk, network
    linked_from: checkout-error-rate-alert   # the actual page

# The alert that pages is the symptom, with the
# resource dashboard one click away in the runbook

What we're doing: Show one incident where a CPU-only alert both fires falsely and misses the real problem.

incident-timeline.txttext

Friday, peak traffic:
09:00 CPU alert fires: cpu_percent > 85%
      -- on-call checks, checkout is fine,
         service is provisioned to run hot,
         acknowledges and moves on.
09:00-14:00 CPU alert re-fires every 30 minutes
      through the entire lunch rush.
         -- on-call starts ignoring the alert.
14:12 A downstream payment API starts timing
      out. CPU stays at 60% the whole time --
      threads are blocked waiting on the network
      call, not computing.
14:12-14:40 Checkout error rate climbs to 22%.
      No infrastructure alert fires -- CPU
      never crosses 85%.
14:40 A customer support ticket surfaces the
      outage; engineering finds out from a
      human, 28 minutes after it started.

2: This alert had zero user impact and was already routine noise by the time the real incident started.
8: CPU is a poor proxy for this failure mode: blocked-on-network threads do not show up as high CPU at all.
13: The 28-minute detection gap is the direct cost of relying on an infrastructure metric that had no causal link to this particular failure.

Why this works: The same incident illustrates both failure modes of infrastructure-noise alerting at once: routine false pages trained the on-call engineer to ignore the alert (alert fatigue), and the actual outage never crossed the monitored threshold at all, so nothing paged on it.

Treating "no infrastructure alert fired" as proof nothing is wrong

Wrong

text

# Incident review note
"CPU, memory, and disk were all nominal
throughout the incident window, so this
was not a capacity issue" -- investigation
closed without checking error rate or
latency dashboards at all.

Better

text

# Incident review note
"CPU, memory, and disk were nominal, but
checkout error rate hit 22% and p99 latency
tripled during the same window -- root
cause was thread starvation waiting on a
slow downstream dependency, not resource
exhaustion. Action: add a symptom alert on
checkout error rate; keep CPU as context only."

What you see: An incident review closes with "infrastructure looked fine" as the conclusion, when infrastructure metrics were never capable of detecting this class of failure in the first place — the team walks away having learned nothing, and the same failure mode recurs because nothing that would catch it next time was added.

Why: This is a distinct mistake from picking infrastructure noise as the alert trigger in the first place (the example above) — it is the reasoning failure that follows from it: using the absence of an infrastructure alert as evidence of health, rather than checking the symptom metrics directly. A clean resource dashboard only proves resources were not the bottleneck; it says nothing about whether users were affected.

Where a raw resource metric belongs versus where an alert should trigger from

Infrastructure-noise alerting

+Pages on CPU/disk/memory crossing a fixed threshold
+No universal threshold maps to user impact
+Pages during normal, healthy high-load periods
+Misses incidents where the bottleneck is not resource-shaped

Symptom-based alerting

−Pages on error rate, tail latency, queue age, availability
−Directly tied to what a user or dependent system feels
−Infrastructure metrics stay as linked diagnostic dashboards
−Catches non-resource-shaped incidents (locks, slow dependencies)

Infrastructure-noise alerting versus symptom-based alerting for the same incident

Infrastructure-noise alerting versus symptom-based alerting for the same incident
Property	Infrastructure-noise alert	Symptom-based alert
What it measures	A raw resource level (CPU%, disk%, memory%)	What the user or dependent system actually experiences
False-positive risk	High — many high-resource states are perfectly healthy	Low — the alert only fires when something is actually broken
False-negative risk	High — real incidents can occur with resources nowhere near threshold	Low, if the symptom set is complete (error rate, latency, saturation, queue age, freshness, availability)
Best use	Diagnostic context once a symptom alert has fired	The primary trigger that pages someone

Remember: A raw resource reading (CPU, disk, memory) has no fixed threshold that maps to user impact, so it makes a poor primary alert: it pages during healthy high-load periods and stays silent through incidents that are not resource-shaped (locks, slow dependencies, connection exhaustion). Keep infrastructure metrics as saturation signals and runbook-linked diagnostic dashboards; let the page itself come from a symptom.

Operational health checks

Liveness vs readiness and the restart-vs-rotation split, plus synthetic monitoring as an external, scheduled complement.

Health checks: readiness vs liveness, and synthetic monitoring

coreintermediate

A liveness check answers one narrow question: "is this process still running and not permanently stuck?" If it fails, the correct response is to restart the process — it is not coming back on its own. A readiness check answers a different question: "is this process currently able to serve traffic right now?" A process can be alive (not stuck, not crashed) while not ready — still loading a cache on startup, temporarily disconnected from its database, or deliberately draining connections before a shutdown. Restarting on a readiness failure would be wrong (the process is fine, it just needs a moment or a dependency to recover); routing traffic to it would also be wrong (it cannot serve requests correctly yet). Orchestrators like Kubernetes act on these differently: a failed liveness probe triggers a restart, a failed readiness probe removes the instance from the load balancer's rotation without restarting it. Synthetic monitoring is a third, complementary check: rather than asking one instance about itself, an external agent periodically performs a real user-like action (log in, load a page, complete a checkout) from outside the system, on a schedule, whether or not any real user happens to be doing so at that moment — catching failures that internal health checks cannot see, such as a broken DNS record, an expired TLS certificate, or a working backend behind a broken load balancer rule.

Think of it as

Think of liveness as "does this employee have a pulse" — a basic check where the only correct response to "no" is calling an ambulance (restart). Readiness is "is this employee at their desk, logged in, and ready to take the next customer" — an employee can be alive and well but on a lunch break, in which case the right move is to route the next customer to someone else, not to fire them. Synthetic monitoring is a mystery shopper: someone who is not an employee walks in on a fixed schedule, tries to buy something exactly the way a real customer would, and reports back — catching problems like "the front door is locked" that no amount of asking employees "are you okay?" would ever reveal.

yaml

# Kubernetes probe configuration (conceptual)
livenessProbe:
  httpGet:
    path: /healthz        # "am I stuck?" -- cheap, no dependency checks
    port: 8080
  periodSeconds: 10
  failureThreshold: 3       # restart after 3 consecutive failures

readinessProbe:
  httpGet:
    path: /readyz          # "can I serve traffic?" -- checks real deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 2       # pulled from rotation after 2 failures

What we're doing: Show why a single combined health endpoint causes both a restart storm and a wrongly-live instance.

combined-healthz-incident.txttext

1. Service exposes one endpoint, /healthz, used
   for BOTH liveness and readiness probes.
2. /healthz checks: process responds AND database
   connection succeeds AND cache is warm.
3. The database has a 90-second maintenance
   failover. /healthz starts failing for every
   instance at once, cluster-wide.
4. Kubernetes reads the failure on the LIVENESS
   probe and restarts every instance -- even
   though none of them were actually stuck.
5. Restarted instances now re-enter their startup
   sequence, re-warming caches, during the exact
   window the database is still failing over.
6. The restart storm outlasts the original 90s
   database blip by several minutes.

2: Bundling a dependency check (database, cache) into what is read as a liveness probe is the root cause -- a dependency being briefly unavailable is a readiness concern, not a liveness one.
4: This is the wrong response: restarting a perfectly healthy process does nothing to fix an external database failover, and destroys the warm cache state that was fine seconds earlier.
6: The self-inflicted restart storm is now the dominant outage, worse and longer than the database blip that triggered it.

Why this works: Splitting the single endpoint into a cheap liveness check (process responds) and a separate readiness check (dependencies healthy) means the same database blip would have simply pulled instances from rotation for 90 seconds — no restarts, no re-warming, and traffic resumes the moment the database recovers.

Making the liveness probe expensive enough to fail under load, not just when actually stuck

Wrong

yaml

livenessProbe:
  httpGet:
    path: /healthz   # runs a full DB query
                     # and checks 3 downstream APIs
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Better

yaml

livenessProbe:
  httpGet:
    path: /healthz   # only checks: is the event
                     # loop / request handler
                     # responding at all?
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3   # more tolerant of transient blips

What you see: Under normal peak load, the liveness probe itself starts timing out (because it competes for the same thread pool and downstream connections as real traffic), triggering restarts of instances that were never actually stuck — the restarts then reduce total capacity, pushing remaining instances further into the same overload, restarting more of them in turn.

Why: This is a different way the liveness/readiness distinction gets broken than the combined-endpoint example above: even a probe correctly wired only to liveness can cause a self-inflicted cascade if it is expensive (real DB queries, downstream calls) rather than a cheap check of whether the process itself is responsive. A liveness probe should be nearly free to answer when the process is healthy — anything that makes it slow under load turns "high load" into "probe times out" into "restart," which is a death spiral entirely of the probe's own making.

A pod's health across startup, steady state, and a dependency blip

Starting

start

Alive + Ready

Alive, not Ready

Restarted

end

The three operational checks and what triggers on each

The three operational checks and what triggers on each
Check	Question it answers	What happens on failure
Liveness	Is the process stuck or crashed?	Orchestrator restarts the process/container
Readiness	Can this instance serve traffic right now?	Instance removed from load-balancer rotation, not restarted
Synthetic monitoring	Can a real user complete a real action, end-to-end, right now?	Alert fires — catches DNS, TLS, and routing failures no internal check sees

Remember: Liveness asks "is the process stuck?" and its failure means restart; readiness asks "can this instance serve traffic right now?" and its failure means pull-from-rotation without restarting. Never bundle dependency checks into the liveness probe, and keep the liveness check itself cheap. Synthetic monitoring is a third, complementary layer — a scheduled, external, real-user-like check that catches DNS, TLS, and routing failures no internal probe can see.