Operational Excellence

Runbooks and incident response, automating repeatable work, and readiness before go-live

Runbook ≠ playbookknown task vs unknown problem
Mitigate first, diagnose afterthe outage lasts as long as the investigation
A precise runbook is nearly a scriptautomate the mechanism, keep the decision
Documented, measurable, testablea dashboard is not an answer

1.Runbooks, Playbooks, Incidents, and Postmortems

A runbook is the procedure for a task you know how to do — restart the service, rotate the credential, fail over the database. A playbook is the procedure for investigating something you do not yet understand. Incident management is the process that runs during the outage, and the postmortem is what turns the outage into a change that prevents the next one.

Key features

AWS defines operational excellence as the ability to support development and run workloads effectively, gain insight into their operations, and continuously improve supporting processes and procedures to deliver business value
A runbook step is executable as written — a specific command with specific arguments, not "check the database"
A playbook is a decision tree: symptom, what to look at, what each result means, and where to go next
Incident roles matter more than incident tooling: someone coordinates, someone investigates, someone communicates, and one person holds the decision
A postmortem produces owned actions with dates; without those it is a document nobody reads twice
Blameless does not mean causeless — the goal is to change the system so an ordinary mistake cannot cause the same outcome, which is the continuous-improvement half: findings become runbook edits, alarms, and automation

Runbook or playbook?

Runbook or playbook?
Situation	Document	Shape
Rotate the database credential	Runbook	Numbered steps, exact commands, expected output
Fail over to the secondary Region	Runbook	Steps plus a decision point for who authorises it
Latency is high and nobody knows why	Playbook	A decision tree over the observable signals
A customer reports data that looks wrong	Playbook	What to check, in what order, and what each result implies
Scale the cluster for a known event	Runbook	Preferably automated, with the runbook as the fallback

The loop operational excellence is actually describing

pagesguided byfollowed byproducesimproves

Detect

an alarm, not a customer

Respond

roles, playbook, runbook

Resolve

mitigate first, fix second

Postmortem

owned actions with dates

Improve

alarms, runbooks, automation

Signature

runbook (known task, exact commands) · playbook (unknown problem, decision tree) · mitigate → diagnose → postmortem → change

Runbook = known task, executable as written. Playbook = unknown problem, a decision tree over signals. During an incident, mitigate first and diagnose later. Postmortems produce owned, dated actions — that is the loop AWS means by "continuously improve".

2.Automating Repeatable Operational Work

Any operational task performed the same way more than a few times is a candidate for automation. The gain is not only time saved — it is that the automated version is consistent, auditable, and does not depend on whoever happens to be on call knowing the trick.

Key features

Automation removes variance first and time second — the same task done by five people five ways is the real problem
The natural progression is manual → documented runbook → script run by a human → triggered automatically, and each step is useful on its own
Automate the mechanism before the decision: a one-button failover is safer than an automatic one, and far safer than a manual sequence
Systems Manager Automation documents are the AWS-native place for operational runbooks that execute, and can be triggered by an alarm or an event
Automation itself needs the same treatment as application code: version control, review, and least-privilege permissions
Anything that runs unattended needs an alarm on its failure, or a broken automation is silently no automation at all

What to automate first

What to automate first
Task	Why it is a good first candidate	Trigger
Restore-test a backup	Frequent, mechanical, and validates something important	Scheduled
Stop non-production out of hours	Purely mechanical, immediate saving	Scheduled
Expire old snapshots and images	Unbounded growth otherwise	Scheduled / lifecycle policy
Rotate a credential	Error-prone by hand, security-relevant	Scheduled (Secrets Manager rotation)
Remediate a compliance finding	Well-defined fix for a well-defined state	AWS Config remediation
Fail over	Automate the steps; keep the decision human	One button, invoked by a person

Command

text

# Automation is code, so it gets what code gets
- version control and review
- a least-privilege role, not an administrator one
- an alarm on failure, and on not-running-at-all
- a documented manual fallback for when it is broken

Automate for consistency first, time second. Progress manual → runbook → script → triggered, and automate the mechanism before the decision. Treat automation as code — version control, least privilege, and an alarm on both failure and on not having run at all.

3.Operational Readiness and Change Management

Operational readiness is the question "can this be run in production", asked before it is. The answer has three parts: the procedures exist and are written down, there are numbers that say whether it is healthy, and both can be tested rather than assumed. Change management is the same discipline applied to every subsequent change.

Key features

Documented means executable by a non-expert — the test is having someone unfamiliar follow it, not having someone knowledgeable review it
Measurable means there is an alarm, not a dashboard: something that fires without a human watching
Testable means the procedure has been run in a rehearsal, which is the only source of a real duration
A readiness review before go-live is the natural milestone, and AWS treats go-live as one of the architecture milestones for exactly that reason
Change management is readiness applied continuously: every change should say how it is deployed, how it is observed, and how it is reversed
Deployment safety is part of this — a change with no rollback path is a change that cannot be made safely, regardless of how well it was reviewed

A go-live readiness checklist that is worth the meeting

A go-live readiness checklist that is worth the meeting
Question	Evidence, not opinion	Failing answer
Who is paged, and how?	A rotation exists and a test page was received	"The team channel"
What do they read when paged?	A linked runbook per alarm	"They will figure it out"
How do we know it is healthy?	Alarms on user-visible signals	A dashboard nobody watches
How is it deployed?	A pipeline run end to end at least once	"Manually for now"
How is it rolled back?	A rollback performed in staging	"Redeploy the old version"
What is the recovery procedure?	A restore or failover actually executed	"We take backups"

Three properties, each meaningless without the one below

Testable

The procedure has been rehearsed and timed

Measurable

An alarm fires without a human watching

Documented

A non-expert can follow it

Exists

Somebody has thought about how this is operated at all

Signature

documented + measurable + testable · every change: how deployed, how observed, how reversed

Documented (a non-expert can follow it), measurable (an alarm fires unattended), testable (it has been rehearsed). Ask them at go-live and on every change: how is it deployed, how is it observed, how is it reversed. A dashboard is not an answer to the second.

Spotted a mistake on this page? Tell us.

Read Sheet