beginnerintermediateadvanced

System Design · Section 63

SRE and Operational Thinking

Level: intermediate
Read: 16 min
Concepts: 2

SLOs and error budgets (covered in full in §8) decide whether a system currently warrants action — this section is everything downstream of that decision, and everything that prevents needing it as often. Incident response gives an active outage a named incident commander and a mitigate-before-root-cause order of operations; on-call is the paged rotation with an explicit escalation path so no page is silently dropped; postmortems are blameless by convention, naming the systemic gap rather than the engineer, because blame teaches people to hide information instead of surfacing it. Capacity planning and change management are the preventive half — forecasting headroom ahead of demand, and gating how changes reach production, since a recent change is the single most common cause of incidents. Operability itself is then a set of concrete design properties rather than a process layered on afterward: canary or blue-green releases limit a bad deploy's blast radius, but a rollback is only genuinely safe if the data layer stayed backward-compatible; graceful degradation requires deciding in advance which functionality is core and which can be shed under stress; and clear, named ownership is what gets the right rotation paged automatically instead of an incident sitting unactioned.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

SLOs/error budgets decide whether to act (§8) — this section covers incident response, on-call, postmortems, capacity planning and change management, the parts that happen next.
Mitigate the user-facing symptom before root-causing, under a named incident commander — restoring service is faster and takes priority over understanding why.
Blameless postmortems target the systemic gap, not the person — a name in the root cause teaches people to hide information, not to make fewer mistakes.
A rollback is only safe if the data layer is backward-compatible — canary/blue-green limit exposure, but a one-way schema or format change can make "revert the code" impossible.
Graceful degradation and clear ownership are design-time decisions — deciding what is core versus shed-able, and who is paged by name, cannot be improvised mid-incident.

What you will be able to do

Distinguish SLOs/error budgets (the "should we act" signal) from incident response, on-call, postmortems, capacity planning and change management (what happens once you do)
Run an incident with a clear commander and a mitigate-before-root-cause order of operations
Write a blameless postmortem whose action items target a systemic gap rather than an individual
Design deploys and rollbacks so a revert stays genuinely possible, and decide in advance what a system should gracefully shed under stress

The operational-maturity vocabulary beyond SLOs

Incident response, on-call, postmortems, capacity planning and change management — what happens once an SLO says act, and what prevents the next incident.

Incident response, on-call, postmortems, capacity planning and change management

coreintermediate

SLOs and error budgets (covered in full elsewhere) tell you whether something is currently broken enough to act on — the rest of operational maturity is what happens once you decide to act, and what you do afterward so the same failure gets less likely. Incident response is the structured process for restoring service during an active outage: someone is explicitly in charge (an incident commander), status is communicated on a fixed cadence, and mitigating the symptom comes before finding the root cause. On-call is the rotation of engineers carrying paging responsibility for a service outside business hours, with an explicit handoff and escalation path so no page is silently dropped. A postmortem is the written record produced after an incident is resolved — blameless by convention, meaning it names the systemic and contributing factors rather than the person who happened to be holding the pager, because blame teaches people to hide information rather than surface it. Capacity planning is forecasting resource needs ahead of demand (traffic growth, seasonal peaks, a marketing launch) so scaling is a scheduled decision rather than a reactive scramble. Change management is the discipline of controlling how changes (deploys, config, infrastructure) enter production — the biggest source of incidents in most systems is a recent change, so the process that gates changes is itself a reliability lever, not just paperwork.

Think of it as

Picture a hospital emergency room paired with its morbidity-and-mortality conference. The ER (incident response) has one attending physician calling the shots, a triage order (stop the bleeding before ordering the biopsy), and a shift schedule (on-call) so someone is always reachable. After the patient stabilizes, the M&M conference (postmortem) reviews the case with no one on trial — the question is "what about our protocol let this happen," not "whose fault was it" — because a resident who fears blame stops reporting near-misses. Capacity planning is the hospital's bed-forecasting for flu season, done in October, not the night the ER is already full. Change management is the hospital's own credentialing and surgical-checklist process — most preventable harm traces back to a procedure that skipped a step, so the checklist is a patient-safety mechanism, not bureaucracy.

text

Incident timeline (mitigate before root-cause):
detect -> declare incident -> assign IC -> mitigate
       -> service restored -> postmortem -> action items

On-call escalation:
page primary -> no ack in 5 min -> page secondary
             -> no ack in 5 min -> page team lead

What we're doing: Trace one incident from a paged alert through to postmortem action items.

incident-timeline.txttext

1. 02:14 - Page fires: checkout error rate above
   threshold (this threshold is an SLO burn-rate
   alert, defined and tracked elsewhere).
2. 02:16 - On-call engineer acknowledges; declares
   an incident and self-assigns as IC since no one
   else is online yet.
3. 02:19 - IC posts in the incident channel: "Sev2,
   checkout errors, investigating."
4. 02:24 - IC identifies the last deploy (18 minutes
   before the page) as the likely cause and rolls
   it back rather than debugging the new code live.
5. 02:27 - Error rate returns to baseline. IC posts
   "mitigated," keeps the incident open to confirm
   stability.
6. 02:45 - Incident closed. IC schedules a postmortem
   for the next business day.
7. Two days later - postmortem published: root cause
   was a missing null check shipped without a canary
   stage; action item is to require canary rollout
   for this service, not to name the author.

4: Rollback (mitigation) happens before anyone understands the actual root cause -- restoring users takes priority over debugging live.
9: The escalation path did not need to fire here, but the incident could have named a second on-call automatically if step 2 had gone unacknowledged.
17: The action item targets the missing safety net (no canary stage), not the person who wrote the bug -- this is what "blameless" means in practice.

Why this works: The five terms this concept covers are not abstract definitions here — incident response is steps 2-6, on-call is who got paged in step 1 and who would have been paged next, and the postmortem in step 7 is where the fix becomes systemic instead of "don't do that again."

Debugging the root cause live before mitigating the user-facing symptom

Wrong

text

# IC spends the first 40 minutes of the incident
# reading application logs and reproducing the bug
# locally, while checkout stays broken for users
def on_incident():
    investigate_root_cause()   # slow, thorough
    fix_and_deploy_the_real_fix()
    # rollback never considered

Better

text

def on_incident():
    if recent_deploy_exists():
        rollback(recent_deploy)   # mitigate first
    elif failover_available():
        failover()
    # root-cause investigation happens after
    # service is restored, feeding the postmortem

What you see: Users experience a much longer outage than necessary because the team optimized for understanding the problem instead of ending it — by the time the real root cause is found, the same fix (a rollback) that was available in minute one is what actually resolves it, just an hour later.

Why: Root-causing under active user impact is slower and riskier than reverting to a known-good state, and every extra minute of investigation is a minute the incident stays open. The mitigate-first convention exists precisely so that "why did this happen" is answered on a timeline that does not extend the outage.

An incident from detection to postmortem action items

Detected

start

Declared

Mitigated

Resolved

end

Postmortem written

end

The five operational-maturity terms this concept covers, and what each answers

The five operational-maturity terms this concept covers, and what each answers
Term	Question it answers	Key artifact or mechanism
Incident response	Who is in charge right now, and what do we do first?	Incident commander, status updates, mitigate-before-root-cause
On-call	Who gets paged, and what if they don't answer?	Rotation schedule + escalation policy
Postmortem	What let this happen, and what do we change?	Blameless written report with action items
Capacity planning	Will we have enough headroom before we need it?	Demand forecast vs. current + planned capacity
Change management	How does a change earn its way into production safely?	Review, staged rollout, and a documented rollback path

Remember: SLOs and error budgets decide whether to act — this concept is everything that happens once you do: an incident commander mitigates before root-causing, on-call has an explicit escalation path so no page is silently dropped, postmortems are blameless and target systemic gaps, and capacity planning plus change management are the preventive half that heads off saturation and change-caused incidents before they start.

Operability as a design property, not an afterthought

Safe deploys, rollbacks that stay real, graceful degradation decided ahead of time, and clear, named ownership.

Designing for operability: safe deploys, rollbacks, graceful degradation and clear ownership

coreintermediate

Operability is not something you bolt on with a runbook after the system is built — it is a set of design properties the architecture either has or does not. A safe deploy limits the blast radius of any single release: canary (route a small slice of traffic to the new version, verify, then expand) and blue-green (run two full environments, switch traffic between them) both exist so that a bad release affects a fraction of users, or none, rather than everyone at once. A rollback is only "safe" if the system was designed to make it fast and low-risk — that means avoiding one-way migrations (a schema change or message format shift that the old code cannot read) so reverting the code is not blocked by data that has already moved forward. Graceful degradation is a system shedding non-critical functionality under stress while preserving its core purpose — a shopping site that disables recommendations to keep checkout working, rather than an all-or-nothing outage. Clear ownership means every service has a specific team or on-call rotation actually responsible for it — an "orphaned" service nobody owns is where incidents linger longest, because no one has the standing authority or context to act on it.

Think of it as

Think of a building's fire safety design, not its fire drill. A canary deploy is like opening one floor of a newly renovated building to occupants before opening the rest — if smoke alarms go off, only that floor evacuates. Blue-green is having two complete parallel elevators shafts and moving all traffic from one to the other; you can move it back with a single switch, not construction. A safe rollback the same way needs the building's water and power to still work after switching back to the old shaft — if the renovation reran conduits in a way only the new shaft understands, "switch back" is not actually possible anymore. Graceful degradation is a building whose fire system cuts non-essential power (decorative lighting, elevators) but keeps emergency lighting and exit doors powered — it degrades on purpose, in a chosen order, instead of going fully dark. Clear ownership is simply which building manager's number is on the sign by the fire panel — a building with no listed manager is the one where the alarm rings the longest before anyone responds.

text

Canary rollout:
deploy v2 to 5% -> watch error rate/latency for N min
  -> healthy? expand to 25% -> 50% -> 100%
  -> unhealthy? auto-rollback to v1 at current %

Graceful degradation (circuit breaker):
call recommendations_service() with timeout
  -> success: render recommendations
  -> timeout/error: render page without them
     (checkout path is untouched either way)

What we're doing: Trace a risky release through a canary rollout, an automatic rollback, and the degradation path that kept the core product usable in the meantime.

safe-rollout.txttext

1. New checkout-service version deploys to 5% of
   traffic (canary), old version still serves 95%.
2. Canary's error rate climbs to 4x baseline within
   two minutes -- recommendations sub-call is timing
   out against the new version's changed API.
3. Automated rollback reverts the canary to the old
   version; the 95% on old code was never affected.
4. Separately, checkout's circuit breaker had already
   marked the recommendations dependency unhealthy and
   was serving checkout pages without recommendations
   -- so the 5% canary traffic still completed
   purchases throughout the incident.
5. On-call for checkout-service (a named rotation,
   not "whoever notices") is paged automatically by
   the canary health-check failure in step 2.
6. Postmortem action item: the new version's API
   change should have been backward-compatible with
   the recommendations client, caught by contract
   tests before canary.

4: Only 5% of traffic was ever exposed to the regression -- this is the entire point of canarying instead of deploying to 100% directly.
13: Graceful degradation (the circuit breaker) meant even the affected 5% kept completing checkouts -- degrading a non-critical feature protected the core purpose.
17: A named on-call rotation being paged automatically is what clear ownership looks like in practice, not a person happening to notice a dashboard.

Why this works: None of these four properties helped in isolation -- the canary limited exposure, the circuit breaker preserved the core purchase flow during that exposure, and named ownership meant the regression was paged and reverted in minutes instead of discovered by users first.

Coupling a schema migration to the same release as the code that requires it

Wrong

text

# Same deploy: drop old column AND ship code
# that no longer reads it
ALTER TABLE orders DROP COLUMN legacy_status;
# v2 code reads only the new 'status' column
# v1 code (needed for rollback) still expects
# legacy_status to exist

Better

text

# Release 1: add new column, backfill, dual-write
# both columns; v1 code untouched, v2 code can read
# either. Deploy and verify this is safe first.
ALTER TABLE orders ADD COLUMN status;
# backfill status from legacy_status; dual-write

# Release 2 (later, separate deploy): only after
# rollback of release 1's code is no longer needed,
# drop legacy_status

What you see: The canary for this release looks fine on its own metrics, but the moment anyone needs to roll back to the previous code version -- for this release or a completely unrelated one deployed after it -- the old code crashes immediately because the column it expects no longer exists. The team discovers, mid-incident, that "rollback" was never actually possible.

Why: A safe deploy and a safe rollback are the same design problem looked at from two directions: if the data layer changes in a way only the new code understands, the rollback path is fictional no matter how good the canary or blue-green mechanics are. Backward-compatible, staged schema changes (expand, backfill, dual-write, then later contract) keep the old code runnable for as long as it might need to be rolled back to.

A canary rollout with an automatic rollback path

5% canary

start

Expanding

Fully rolled out

end

Rolled back

end

Canary vs. blue-green as two different ways to make a deploy safe

Canary vs. blue-green as two different ways to make a deploy safe
Property	Canary	Blue-green
Exposure	Small % of traffic first, then ramps up	All traffic switches at once
Rollback speed	Slower — traffic must be shifted back down	Instant — flip the router back to the old environment
Infra cost	Lower — new version runs alongside old at partial scale	Higher — two full-scale environments run simultaneously
Best for	Catching regressions before they reach everyone	Changes where any partial-exposure risk is unacceptable

Remember: Operability is designed in, not layered on after: canary/blue-green limit a bad release's blast radius, but a rollback is only real if the data layer stayed backward-compatible; graceful degradation requires deciding in advance what is core versus shed-able, not during the incident; and clear ownership means a specific rotation is paged by name, not "whoever notices."