Monitoring and Production Operations

Key features

• Metrics, structured/JSON logs, and distributed tracing (OpenTelemetry) are taught in depth in Logging and Observability — this section assumes that ground
• An alert is a threshold rule evaluated against a metric — no metric, no alert
• A dashboard visualizes metrics over time — it has nothing to chart without them
• An incident responder reads logs for what happened and traces for where time went — the diagnostic raw material
• SLIs (next concept) are just metrics chosen specifically to represent user-facing reliability

Signature

metrics / logs / traces → alerts, dashboards, SLOs, incident response

Key features

• Liveness vs. readiness, the Kubernetes probe vocabulary, and the Python check functions are taught in full in Logging and Observability
• A readiness check is often the fastest signal that an SLI (next concept) is about to be breached — traffic is already being turned away
• Health-check endpoints are what an uptime/synthetic monitor polls to feed an alert or dashboard
• Incident response usually starts by hitting the health-check endpoint before reading logs or traces

Signature

GET /healthz, GET /readyz

Key features

• An alert fires on a condition over a time window (e.g. error rate > 5% for 5 minutes) — not a single bad data point, which pages on noise
• A good alert is actionable: the SRE book's standard is that every page should require a human response, or it should not page at all
• A dashboard answers "what is the current state" at a glance; it does not wake anyone up on its own
• Alerting on symptoms users feel (latency, error rate) is preferred over alerting on every possible internal cause — fewer, higher-signal alerts
• Alert fatigue — too many low-value pages — is a named failure mode; it trains responders to ignore alerts, including real ones

Command

# The shape shared by Prometheus Alertmanager, CloudWatch Alarms,
# Grafana alerting, and similar tools:
IF <metric> <comparison> <threshold> FOR <duration>
  THEN notify(<channel>)  # page, Slack, email — by severity

# Example rule, in words:
IF rate(http_requests_failed) / rate(http_requests_total) > 0.05
   FOR 5m
   THEN page(on-call)

Key features

• SLI (Service Level Indicator): a measured metric — e.g. the proportion of successful, fast-enough requests
• SLO (Service Level Objective): the target for an SLI over a window — e.g. "99.9% of requests succeed, over 30 days"
• Error budget: 100% minus the SLO — the amount of unreliability allowed before the SLO is breached (99.9% SLO leaves a 0.1% budget)
• An SLA (Service Level Agreement) is an SLO with a consequence attached (a refund, a penalty) — not covered by the roadmap here, but the term a reader will see next to these
• Google's SRE book treats the error budget as a shared decision tool: budget remaining favors shipping features, budget exhausted favors a change freeze

Command

# SLI: measure it
success_rate = successful_requests / total_requests

# SLO: the target you set for that SLI, over a window
slo_target = 0.999   # 99.9% over 30 days

# Error budget: what's left before the SLO is breached
error_budget = 1 - slo_target
budget_remaining = error_budget - (1 - success_rate)

Key features

• Incident response prioritizes mitigation over diagnosis — restoring service now beats understanding the full cause immediately
• A named Incident Commander owns coordination during a major incident, separate from whoever is actually fixing it — one person decides, many can investigate
• Root cause analysis is rarely one root cause — the SRE book favors "contributing factors" over a single cause, since most outages need several things to go wrong together
• The "Five Whys" technique repeatedly asks why until reaching a systemic, fixable condition — not a single person's mistake
• A rollback (next concept) is very often the fastest mitigation during incident response, even before the root cause is known

Command

Incident lifecycle:  detect -> declare -> mitigate -> resolve -> analyze
Root cause analysis: "Five Whys" — ask why repeatedly past the first answer
                      until you reach a systemic, fixable condition

Key features

• Blameless: the document names contributing factors and broken systems, never "who caused it" — a named practice from Google's SRE book, not just a courtesy
• A postmortem is written after root cause analysis (the previous concept) — it is where that analysis is recorded, not where it happens
• Standard sections: summary, impact (duration, users affected), timeline, root cause / contributing factors, what went well, action items with owners
• Action items need an owner and often a deadline — a postmortem with no follow-through just repeats the same incident later
• A postmortem criterion is often written down in advance (e.g. "any incident that breaches the SLO gets one") so writing one is not a political decision made after the fact

Command

Postmortem sections (SRE book shape):
  Summary        — one paragraph, what happened
  Impact         — duration, users/requests affected, SLO/error-budget impact
  Timeline       — detect -> declare -> mitigate -> resolve, with timestamps
  Root cause     — contributing factors (see Incident response / RCA)
  What went well — genuinely, not just criticism
  Action items   — each with an owner and a status

Key features

• The core pattern is try/except around a dependency call, with a fallback return value in the except branch
• Catch the SPECIFIC exception the dependency raises — a bare except Exception also swallows real bugs in your own code, not just the outage
• A circuit breaker stops CALLING a failing dependency after N consecutive failures, instead of retrying (and timing out) on every request
• A circuit breaker has to reset (close) eventually, or a temporary outage becomes a permanent one for that dependency
• Graceful degradation trades completeness for availability — a recommendations widget can go blank; checkout usually cannot

Command

def get_data():
    try:
        return call_dependency()
    except SpecificDependencyError:   # not a bare except
        return fallback_value

Key features

• A rollback reverts the deployed version, not the data — a migration that already ran does not automatically un-run on rollback
• Rolling back is usually faster than a forward fix, because "make it match yesterday" needs no new code review
• A feature flag can act as a fast, partial rollback — turn the new code path off without redeploying anything
• An automated rollback trigger (e.g. error rate over a threshold) can revert before a human even finishes declaring the incident
• Rollback safety depends on backward-compatible database migrations — a rollback that hits a schema the old code cannot read makes things worse

Command

def choose_active_version(current: str, previous: str, errors: int, total: int) -> str:
    if error_rate_breached(errors, total):
        return previous   # roll back
    return current