beginnerintermediateadvanced

AWS · Section 52

Operational Excellence

Level: advanced
Read: 25 min
Concepts: 3

AWS defines operational excellence as supporting development and running workloads effectively, gaining insight into their operations, and continuously improving the supporting processes. In practice that starts with two different documents: a runbook, which is the procedure for a task you already know how to do and must be executable as written by someone unfamiliar, and a playbook, which is a decision tree for a problem nobody understands yet. During an incident the order is mitigate first and diagnose afterwards — the outage otherwise lasts as long as the investigation — and the postmortem closes the loop by producing owned, dated actions rather than sentiments. A precise runbook is most of the work of a script, which is why the natural progression is manual, then documented, then scripted, then triggered; automate the mechanism while keeping the decision human, treat the automation as code with least-privilege permissions, and alarm both on its failure and on its absence. Underneath all of it sits operational readiness: procedures that are documented (a non-expert can follow them), measurable (an alarm fires without anyone watching), and testable (they have actually been rehearsed). Change management is that same discipline applied continuously — every change stating how it is deployed, how it is observed, and how it is reversed.

This section

AWS Well Architected Framework Pillar #1 - Operational ExcellenceBe A Better Dev

What is true here

Runbook = known task with exact commands; playbook = decision tree for an unknown problem.
Mitigate before diagnosing; postmortems produce owned, dated actions.
Automate for consistency first — mechanism automated, decision human — and monitor the automation itself.
Readiness = documented, measurable, testable; a dashboard does not satisfy "measurable".
Every change answers: how deployed, how observed, how reversed.

What you will be able to do

Write a runbook someone unfamiliar can execute, and a playbook that guides an investigation
Run an incident in the right order and produce a postmortem with actionable output
Choose what to automate first, and instrument the automation so its silence is detectable
Run a go-live readiness review that finds what code review cannot
Require a tested reversal for every production change, including the data half

From incident response to something you can hand over

Runbooks, incidents, postmortems

Automate repeatable work

Documented, measurable, testable

Operational Excellence

Runbooks and playbooks, incident response and postmortems, automating repeatable work, and the readiness questions that make a service operable.

Runbooks, Playbooks, Incidents, and Postmortems

coreintermediate

A runbook is the procedure for a task you know how to do — restart the service, rotate the credential, fail over the database. A playbook is the procedure for investigating something you do not yet understand. Incident management is the process that runs during the outage, and the postmortem is what turns the outage into a change that prevents the next one.

Think of it as

Runbook: I know what to do, here are the steps. Playbook: I do not know what is wrong, here is how to find out. Confusing the two produces documents that answer neither question, which is why most "runbooks" are unusable during an actual incident.

What we're doing: See why a well-written document is still unusable at 3am.

runbook-quality.txttext

Unusable runbook:
  "Verify the database is healthy. If not, perform a failover. Notify
   stakeholders."
  -> Which check? Which command? Who are the stakeholders? What counts
     as unhealthy? Every line is a question at the worst moment.

Usable runbook:
  "1. aws rds describe-db-instances --db-instance-identifier prod-orders
      --query 'DBInstances[0].DBInstanceStatus'
   2. If not 'available' for 3 consecutive checks, run the failover in
      step 3.
   3. aws rds reboot-db-instance --force-failover ...
   4. Post in #incident-prod: template at the bottom of this page."

2: This reads well and is a summary, not a procedure. It requires the reader to already know everything the runbook was supposed to supply.
7: Executable as written, by someone who has never done it before. That is the only test that matters.

Why this works: A runbook is read by a tired person under pressure who may not be the expert. Anything requiring judgement they do not have is a gap, and the way to find those gaps is to have someone unfamiliar follow it during a drill.

Writing postmortem actions with no owner and no date

Wrong

text

# Action items: "Improve monitoring." "Consider adding retries."

Better

text

# "Add a CloudWatch alarm on target 5xx rate > 1% for 2 minutes —
#  owner: payments team — due: 12 Sep."

What you see: The same incident recurs six months later and the previous postmortem already listed the fix that would have prevented it.

Why: An action without an owner belongs to nobody and an action without a date is never overdue, so neither ever gets scheduled against competing work. The postmortem is the mechanism that turns an outage into a change, and it only works if its output enters the same queue as everything else.

The loop operational excellence is actually describing

Detect

an alarm, not a customer

Respond

roles, playbook, runbook

Resolve

mitigate first, fix second

Postmortem

owned actions with dates

Improve

alarms, runbooks, automation

Runbook or playbook?

Runbook or playbook?
Situation	Document	Shape
Rotate the database credential	Runbook	Numbered steps, exact commands, expected output
Fail over to the secondary Region	Runbook	Steps plus a decision point for who authorises it
Latency is high and nobody knows why	Playbook	A decision tree over the observable signals
A customer reports data that looks wrong	Playbook	What to check, in what order, and what each result implies
Scale the cluster for a known event	Runbook	Preferably automated, with the runbook as the fallback

Together

text

# Runbook step: executable as written
3. Force a failover and watch the event stream:
     aws rds reboot-db-instance --db-instance-identifier prod-orders \
       --force-failover
   Expected: "Multi-AZ failover completed" within ~120 s.
   If it exceeds 5 minutes, escalate to the database on-call.

Remember: Runbook = known task, executable as written. Playbook = unknown problem, a decision tree over signals. During an incident, mitigate first and diagnose later. Postmortems produce owned, dated actions — that is the loop AWS means by "continuously improve".

Automating Repeatable Operational Work

standardintermediate

Any operational task performed the same way more than a few times is a candidate for automation. The gain is not only time saved — it is that the automated version is consistent, auditable, and does not depend on whoever happens to be on call knowing the trick.

Think of it as

A runbook is automation waiting to happen. Writing the steps down precisely enough to be followed is most of the work of writing the script, which is why good runbooks tend to turn into code.

text

# Automation is code, so it gets what code gets
- version control and review
- a least-privilege role, not an administrator one
- an alarm on failure, and on not-running-at-all
- a documented manual fallback for when it is broken

Automating a task without alarming on the automation

Wrong

text

# A scheduled Lambda cleans up old snapshots. Nobody watches it.

Better

text

# Emit a success metric each run; alarm on failure AND on the absence
# of a run in the expected window

What you see: The automation stopped working four months ago after a permission change, and the first sign is the storage bill or a full disk.

Why: Automation makes work invisible, which is the point and also the risk — nobody notices its absence, because noticing was the job it took over. Alarming on the absence of a successful run is what keeps the invisibility safe.

What to automate first

What to automate first
Task	Why it is a good first candidate	Trigger
Restore-test a backup	Frequent, mechanical, and validates something important	Scheduled
Stop non-production out of hours	Purely mechanical, immediate saving	Scheduled
Expire old snapshots and images	Unbounded growth otherwise	Scheduled / lifecycle policy
Rotate a credential	Error-prone by hand, security-relevant	Scheduled (Secrets Manager rotation)
Remediate a compliance finding	Well-defined fix for a well-defined state	AWS Config remediation
Fail over	Automate the steps; keep the decision human	One button, invoked by a person

Together

text

# The progression, made concrete
manual        : someone remembers to stop staging on Friday
runbook       : a page says exactly which resources and how
script        : one command stops them all
automatic     : a schedule stops them, and alarms if it fails

Remember: Automate for consistency first, time second. Progress manual → runbook → script → triggered, and automate the mechanism before the decision. Treat automation as code — version control, least privilege, and an alarm on both failure and on not having run at all.

Operational Readiness and Change Management

coreadvanced

Operational readiness is the question "can this be run in production", asked before it is. The answer has three parts: the procedures exist and are written down, there are numbers that say whether it is healthy, and both can be tested rather than assumed. Change management is the same discipline applied to every subsequent change.

Think of it as

Three properties, in order. Documented means someone other than the author can operate it. Measurable means you can tell whether it is working without asking anyone. Testable means the claim can be checked rather than believed. A procedure missing any one of them is a hope.

What we're doing: See what a readiness review catches that a code review cannot.

readiness-review.txttext

The service passed code review, security review, and load testing.

Readiness review, one hour before go-live:
  - No alarm exists on the user-visible error rate. There is a dashboard.
  - The on-call rotation covers the team, but nobody has received a test
    page from this service, and the routing was never verified.
  - The rollback is "redeploy the previous image", which has not been
    tried since the pipeline gained a migration step.
  - The runbook links to a Systems Manager document that was deleted
    during a cleanup two months ago.

Four findings, none of which are about the code, and all of which would
have first appeared during an incident.

1: Every earlier review was about whether the software is correct. None of them asked whether it can be operated.
5: Alerting routing is the most common silent failure: the alarm is configured, and nothing checks that the page arrives.
8: Rollback paths decay whenever the pipeline changes, which is why they need re-testing rather than documenting once.

Why this works: Correctness reviews and readiness reviews find disjoint sets of problems. The readiness questions are cheap to ask and only ever get answered honestly before go-live, because afterwards the answer is discovered during an incident.

Accepting "we have a dashboard" as an answer to "how would we know?"

Wrong

text

# Observability: a Grafana board with 24 panels

Better

text

# One alarm per user-visible failure mode, routed to a rotation, with a
# runbook linked from the alarm description

What you see: Incidents are reported by customers, and the review notes that the dashboard clearly showed the problem for the preceding half hour.

Why: A dashboard requires a human to be looking, so it detects nothing outside working hours and nothing during the hours everyone is looking at something else. An alarm is the only part of the setup that acts on its own, which makes "which alarm?" the only version of the question with a real answer.

Three properties, each meaningless without the one below

Testable

The procedure has been rehearsed and timed

Measurable

An alarm fires without a human watching

Documented

A non-expert can follow it

Exists

Somebody has thought about how this is operated at all

A go-live readiness checklist that is worth the meeting

A go-live readiness checklist that is worth the meeting
Question	Evidence, not opinion	Failing answer
Who is paged, and how?	A rotation exists and a test page was received	"The team channel"
What do they read when paged?	A linked runbook per alarm	"They will figure it out"
How do we know it is healthy?	Alarms on user-visible signals	A dashboard nobody watches
How is it deployed?	A pipeline run end to end at least once	"Manually for now"
How is it rolled back?	A rollback performed in staging	"Redeploy the old version"
What is the recovery procedure?	A restore or failover actually executed	"We take backups"

Together

text

# Change management, as three questions on every change
1. How is it deployed?   (the pipeline, not a person)
2. How is it observed?   (which alarm would catch it going wrong)
3. How is it reversed?   (the exact action, already tested)

Remember: Documented (a non-expert can follow it), measurable (an alarm fires unattended), testable (it has been rehearsed). Ask them at go-live and on every change: how is it deployed, how is it observed, how is it reversed. A dashboard is not an answer to the second.