Filter concepts by levelShowing all levels.

AWS · Section 65

AWS Architecture Review Checklist

Level
advanced
Read
45 min
Concepts
6

A review is an audit of a running account, not a reading of a design document, and the difference decides what it finds. The security pass asks six questions — privilege, encryption, secrets, network isolation, logging, vulnerability management — and each is closed by an artefact rather than an assurance, because both sides sincerely believe the assurance and neither has looked recently. The reliability pass is written backwards, starting from failures rather than features: a zone, an instance, the data, the primary, one failed call, a duplicate, an optional dependency, each with the control that answers it and the date the control was last exercised, since a control nobody has run is a plan and plans fail differently from mechanisms. The performance pass refuses to recommend anything before the latency is attributed per hop, which routinely shows that the cache everyone proposed targets a smaller segment than the connection pool nobody mentioned. The cost pass groups the bill by usage type and splits it in two: capacity you chose, which can be shrunk, and architecture that accumulated — NAT data processing, log ingestion, the wrong storage class — which can be removed outright. The operations pass reduces to one question, whether a stranger could recover the system at three in the morning from what is written down, and it is answered by handing the runbook to another team in daylight rather than by asking the author. The sustainability pass finds provisioned capacity doing no work, and work larger than the result it produces. Run in that order, the six passes produce a list of facts with evidence attached instead of a list of opinions.

What is true here

  1. Review the live account: a diagram records intent, and only the account drifts away from it.
  2. Security questions are closed by artefacts — a policy, a key id, a secret ARN, a scan date.
  3. Write reliability as failures with controls beside them, each with the date it was exercised.
  4. Attribute latency and group the bill by driver before recommending anything.
  5. Operability is tested by a stranger following the runbook, never by asking the author.

What you will be able to do

  • Run a six-pass review over an AWS account and produce findings with evidence attached
  • Tell an assurance from an artefact, and ask for the artefact every time
  • Turn a reliability claim into a table of failures, controls and exercise dates
  • Separate cost that can be shrunk from cost that can be removed
  • Test whether a system is operable by someone who did not build it
Six passes over one system, each with its own question

The order matters: a system that is not secure or not reliable does not benefit from being fast or cheap. Each pass produces findings with evidence attached, never an opinion.

  • One system at the top, with six review passes drawn beneath it in two rows of three.
  • Pass 1, Security: who can do what, is it encrypted, where are the secrets, what can reach it, is it logged, when was it scanned.
  • Pass 2, Reliability: name each failure, then the control that answers it and the date it was last exercised.
  • Pass 3, Performance: attribute the latency per hop, then pull the lever that targets the largest segment.
  • Pass 4, Cost: group the bill by usage type and separate chosen capacity from accumulated architecture.
  • Pass 5, Operations: could a stranger recover this at 3 a.m. using only what is written down?
  • Pass 6, Sustainability: what capacity is provisioned and doing no work?
  • Footnote: every finding carries the artefact that proves it, so the output is a list of facts rather than a list of opinions.

AWS Architecture Review Checklist

Six passes over one system — security, reliability, performance, cost, operations, sustainability — each with the questions to ask, the evidence that closes them, and the answer that looks complete and is not.

The Security Pass — Six Questions, Six Pieces of Evidence

coreadvanced

The security pass of a review asks six questions: who can do what, is the data encrypted, where do the secrets live, what can reach this over the network, is every action logged, and when was the code last scanned. Each question is answered with evidence you can point at — a policy, a key id, a security group rule, a scan date — not with a sentence about intent.

Think of it as

Review the account, not the diagram. A diagram shows what someone meant to build; `aws ec2 describe-security-groups` shows what is running. Every one of the six questions has a command or a console page that produces the real answer, and a review that accepts a claim instead of that output finds nothing.

What we're doing: Run the security pass over a design that has already been signed off.

security-pass.txttext
Service: orders-api. Reviewed 2026-09-05. Six questions.

1. LEAST PRIVILEGE
   Claim:    "The task role only has what it needs."
   Evidence: the role carries AmazonDynamoDBFullAccess.
   Verdict:  FAIL. Replace with the four actions the code calls,
             scoped to the two table ARNs it touches.

2. ENCRYPTION
   Claim:    "Everything is encrypted."
   Evidence: RDS uses the AWS managed key; the ALB terminates TLS
             and calls tasks over plain HTTP inside the VPC.
   Verdict:  PARTIAL. At rest passes. In transit stops at the ALB.

3. SECRETS
   Claim:    "Secrets are in Secrets Manager."
   Evidence: the database password is, and a third-party API key
             is a plain value in the task definition.
   Verdict:  FAIL. One secret out of two is not a secrets story.

4. NETWORK ISOLATION
   Evidence: the database group allows 5432 from 10.0.0.0/16.
   Verdict:  FAIL. Allow 5432 from the application security group
             id, so a future subnet cannot inherit access.

5. LOGGING
   Evidence: multi-Region organization trail, log-archive account,
             the workload has no write access to that bucket.
   Verdict:  PASS.

6. VULNERABILITY MANAGEMENT
   Evidence: Inspector enabled; oldest task image built 214 days
             ago; no owner named for the findings queue.
   Verdict:  FAIL. Enabled is not managed.

Two passes, one partial, three fails — on a design that had
already been approved by three people.
3
A managed policy with `FullAccess` in its name is the fastest least-privilege finding there is, and it survives most reviews because the policy name sounds official.
9
Splitting encryption into two questions is what catches the common shape: encrypted at rest, and plain text on the last hop inside the VPC.
15
Secrets are reviewed exhaustively or not at all — one hard-coded key is enough to make the rotation story on the other one irrelevant.
22
A CIDR-based database rule grants access to every future workload placed in that range. A security group id grants it to one thing.

Why this works: Every fail here was invisible in the architecture diagram, because the diagram is a drawing of the intent and each of these is a property of the running account. The six questions are worth asking in this order because they move outward — identity, then data, then credentials, then network, then the record, then the code — and each one has a command that produces an answer nobody can argue with.

Reviewing the diagram instead of the account

Wrong

text
# "The diagram shows the database in a private subnet, so
#  network isolation passes."

Better

bash
# Ask the account, not the drawing.
aws ec2 describe-security-groups --group-ids sg-0db1 \
  --query 'SecurityGroups[].IpPermissions'
# → 5432 open to 10.0.0.0/16, which is every subnet in the VPC.

What you see: A review signs off a design that is safe as drawn, and the account keeps a rule that lets any instance anywhere in the VPC open a database connection — usually discovered when an unrelated team launches something into the same range.

Why: A diagram records what someone intended at the time it was drawn. Configuration drifts away from it through console edits, emergency changes and later additions, and none of those update the picture. Reviewing the live configuration is the only pass that describes the system as it currently is.

Six security questions, and what closes each one

Each question is closed by an artefact, not by an assurance. If the middle column is empty, the question is still open.

  • Six rows, each with a question on the left, the evidence that closes it in the middle, and the failing answer on the right.
  • Least privilege: closed by an Access Analyzer report showing no unused permissions; fails on "we follow least privilege".
  • Encryption: closed by a named KMS key id and a TLS policy; fails on "everything is encrypted".
  • Secrets: closed by a Secrets Manager ARN and a rotation date; fails on a credential in an environment variable.
  • Network isolation: closed by security group rules with no 0.0.0.0/0 on admin or database ports; fails on "it is in a private subnet".
  • Logging: closed by an organization trail in a separate log account; fails on "CloudTrail is on".
  • Vulnerability management: closed by a scan date and a named owner; fails on "Inspector is enabled".

The security pass — what to run, what closes the question

The security pass — what to run, what closes the question
CheckWhat to look atPasses whenFails when
Least privilegeIAM Access Analyzer findings; policies attached to production rolesNo external access finding is unexplained, and no production policy grants `*` on `*`Wildcard actions on production resources, or long-lived user access keys instead of roles
Encryption at restRDS, EBS, S3 and DynamoDB encryption settings, and the key each usesEvery data store names a key, and sensitive stores name a customer-managed keyA store with encryption off, or nobody able to say which key protects it
Encryption in transitLoad balancer listeners, database `require_ssl`, service-to-service callsTLS on every hop that leaves an instance, including the hop behind the load balancerTLS terminated at the edge and plain HTTP the rest of the way
SecretsTask definitions, Lambda environment variables, AMIs, images, repository historyEvery credential resolves from Secrets Manager or Parameter Store at run timeA credential visible in an environment variable, an image layer, or a commit
Network isolationSecurity groups, NACLs, subnet route tables, public IP assignmentData tiers accept traffic only from an application security group, not a CIDR0.0.0.0/0 on an administrative or database port anywhere in the account
LoggingCloudTrail trails, VPC flow logs, load balancer access logs, log destinationsAn organization trail writes to a separate log account the workload cannot alterTrails writing into the same account they audit, or single-Region coverage
Vulnerability managementAmazon Inspector findings, base image ages, patch cadenceA stated cadence, a named owner, and a most-recent scan date inside itScanning enabled with no owner, or a base image nobody has rebuilt this quarter

Together

bash
# The three commands that answer most of the security pass quickly.

# 1. Anything open to the whole internet on an admin or database port?
aws ec2 describe-security-groups \
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 \
  --query 'SecurityGroups[].{Group:GroupId,Name:GroupName}' \
  --output table

# 2. Which trails exist, and are they multi-Region?
aws cloudtrail describe-trails \
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Bucket:S3BucketName}' \
  --output table

# 3. What access has IAM Access Analyzer flagged as reachable from outside?
aws accessanalyzer list-findings \
  --analyzer-arn "$ANALYZER_ARN" \
  --filter '{"status":{"eq":["ACTIVE"]}}' \
  --query 'findings[].{Resource:resource,Type:resourceType}' \
  --output table

Remember: Six questions: privilege, encryption, secrets, network, logging, vulnerabilities. Each is closed by an artefact — a policy, a key id, a secret ARN, a security group rule, a trail in another account, a scan date with an owner. Review the account rather than the diagram, because only one of the two drifts.

See also: least privilege and evaluation · encryption at rest vs in transit · secrets manager and parameter store · centralizing and protecting audit logs · the reliability review

The Reliability Pass — Name the Failure, Then Find the Control

coreadvanced

The reliability pass works backwards from failures rather than forwards from features. Name a thing that can break — a zone, an instance, a disk, a dependency, a duplicate message — and ask which control handles it and when that control was last exercised. A control nobody has tested is a plan, and plans fail differently from mechanisms.

Think of it as

Every reliability control answers exactly one failure. Multi-AZ answers "a zone goes away". Health checks answer "one instance is sick". Backups answer "the data is wrong or gone". Retries answer "the call failed once". Idempotency answers "the retry arrived twice". Line the failures up in a column and the gaps are the rows with nothing beside them.

What we're doing: Turn a reliability claim into a table of failures with controls beside them.

reliability-pass.txttext
Service: orders-api. The claim: "it is highly available".

FAILURE                     CONTROL              LAST EXERCISED
An Availability Zone fails  2 AZs, 4 tasks       never
  → the ASG maximum is 4, so losing a zone leaves 2 tasks
    against a peak that needs 3. The design is multi-AZ and
    the capacity is not. Raise the maximum, or accept the
    degradation in writing.

An instance goes sick       /health returns 200  never
  → the check proves the process is listening. A task with a
    dead database pool passes it and keeps taking traffic.
    Make the check touch the database, and cap it at 500 ms.

The database fails over     RDS Multi-AZ         never
  → the mechanism is real. The application holds a pool of
    connections that will not notice, so the first minute
    after a failover is an outage the drill would have shown.

A charge is submitted twice nothing              n/a
  → the payment call has no idempotency key. Two clicks or
    one retry both charge the customer twice.

The recommendation engine   nothing              n/a
is down
  → the product page throws a 500 rather than rendering
    without recommendations. A non-critical dependency is
    a full outage because nobody decided otherwise.

Five failures. Two controls, both untested. Three gaps.
4
Multi-AZ and multi-AZ capacity are separate properties, and the second one is what an evacuation actually needs.
11
A shallow health check is the most common reliability finding, because it passes every test except the one that matters.
18
The failover mechanism working and the application surviving it are different claims. Only the drill tests the second.
24
Idempotency is the control for duplicates, and duplicates arrive from user behaviour as readily as from infrastructure retries.

Why this works: Starting from failures rather than from features is what makes the gaps visible. A feature list says multi-AZ, health checks and Multi-AZ RDS are all present, and every one of those is true; the failure list shows that the capacity does not survive a zone, the health check does not detect sickness, and nothing at all handles a duplicate charge. The same system, read from the other end, produces a different verdict.

Counting a control that has never been exercised

Wrong

text
# "Multi-AZ RDS is enabled, so database failover is covered."

Better

text
# Force one and measure it.
# aws rds reboot-db-instance --db-instance-identifier orders \
#     --force-failover
# Recovery took 94 s: 38 s of AWS failover, 56 s of the pool
# holding dead connections. The pool setting is the real finding.

What you see: A failover that AWS completes in under a minute produces several minutes of application errors, because the connection pool keeps handing out sockets to an endpoint that has moved.

Why: The managed control and the application behaviour around it are two different systems, and only one of them is AWS's responsibility. Exercising the control is what reveals the second — and it is almost always the second one that dominates the recovery time, because nobody configured it deliberately.

Each failure, its control, and the proof it works

Read left to right. A row with a control and no proof is the row that will surprise you, because the mechanism has never been observed working.

  • Seven rows pairing a failure with the control that answers it and the proof that the control works.
  • A whole Availability Zone fails: answered by multi-AZ across two zones; proved by a zone-evacuation drill.
  • One instance goes sick: answered by a deep health check plus automatic replacement; proved by terminating an instance and watching traffic move.
  • Data is deleted or corrupted: answered by backups with a retention window; proved by a timed restore.
  • The primary database fails: answered by Multi-AZ failover; proved by a forced failover with the application running.
  • A call fails once: answered by bounded retries with backoff and jitter; proved by a fault-injection experiment.
  • A retry arrives twice: answered by an idempotency key; proved by replaying the same request and comparing the result.
  • A non-critical dependency is down: answered by a designed degraded mode; proved by disabling that dependency in a drill.

The reliability pass — the seven checks and what a weak answer looks like

The reliability pass — the seven checks and what a weak answer looks like
CheckThe questionPasses whenWeak answer
Multi-AZWhich tiers span two zones, and does the remaining zone have the capacity?Compute, database and cache each span two zones, sized so one zone can carry the load"It is multi-AZ" for a two-task service where one zone is running at 100%
Health checksDoes the check fail when the instance cannot actually serve a request?The path exercises the database and any critical dependency, and has its own timeoutA `/health` route that returns 200 from the web framework and touches nothing
BackupsWhat is backed up, how far back, and how long does a restore take?Retention is stated, and the last restore has a measured duration and a date"Automated snapshots are enabled" with no restore ever performed
FailoverWhat happens to in-flight connections when the primary is replaced?The application reconnects, the pool recycles, and the drill has been run"RDS handles it" — while the connection pool holds dead sockets for minutes
RetriesHow many, with what backoff, and what stops the retry storm?A bounded count, exponential backoff with jitter, and a circuit breaker"The SDK retries" with the default configuration nobody has looked at
IdempotencyWhat happens when the same request is processed twice?A key derived from the request, stored, and checked before any side effect"Duplicates are rare" — which is a statement about volume, not correctness
Graceful degradationWhich dependencies are optional, and what does the user see without them?Optional dependencies are listed, each with a decided fallback behaviourEvery dependency treated as required, so any one of them is a full outage

Together

text
# One row done properly — the health check, written as a contract.

PATH        GET /internal/health
CHECKS      1. a SELECT 1 against the writer, 250 ms timeout
            2. a Redis PING, 100 ms timeout
            3. queue depth read from a local counter, no network call
RETURNS     200 only when 1 and 2 both pass
            503 otherwise, with the failing check named in the body
TIMEOUT     the whole handler is capped at 500 ms
USED BY     the ALB target group, interval 15 s, unhealthy after 2

# Compare with the usual version:
#   @app.get("/health")
#   def health(): return {"ok": True}
# That returns 200 from a task whose database pool is exhausted,
# so the load balancer keeps sending it traffic it cannot serve.

Remember: List the failures first — zone, instance, data, primary, one bad call, a duplicate, an optional dependency — then write the control beside each and the date it was last exercised. Multi-AZ without spare capacity, a health check that touches nothing, and a backup nobody has restored are all controls that exist only on paper.

See also: multi az baseline · redundancy health checks and replacement · the failure drill catalogue · what idempotency means · the performance review

The Performance Pass — Find the Bottleneck Before Judging the Design

standardadvanced

The performance pass starts with one question: where does the time actually go? Until a latency breakdown exists, every recommendation is a guess. Once it exists, the six levers — compute size, cache, connection pool, database tuning, CDN, and moving work out of the request — sort themselves by which part of the breakdown they shrink.

Think of it as

A request is a budget. Measure how the milliseconds are spent, then apply the lever that targets the largest line. Adding a cache to a request that spends 80% of its time waiting on a connection pool improves nothing, and the review will still record it as an improvement because nobody measured before or after.

text
measure first → attribute the time per hop → pull the lever that targets the largest segment → measure again
Where the milliseconds go, and which lever moves each one

The bar is one request. Each segment has its own lever, and a lever aimed at a small segment cannot produce a large improvement.

  • A horizontal bar representing a 900 millisecond request, split into five segments with the lever for each named underneath.
  • Edge and network, 40 milliseconds: the lever is a CDN and a cache behaviour that does not forward every cookie.
  • Waiting for a database connection, 310 milliseconds: the lever is connection pool sizing or a proxy.
  • The query itself, 260 milliseconds: the lever is an index, a rewrite, or an instance with more memory.
  • Application compute, 130 milliseconds: the lever is right-sizing the compute or the code path itself.
  • Sending an email inline, 160 milliseconds: the lever is a queue, which removes the segment rather than shrinking it.
  • Footnote: adding a cache in front of the query targets 260 of 900 milliseconds, while moving the email out targets 160 and costs nothing to run.

Adding a cache because latency is high

Wrong

text
# "p95 is 900 ms. Put Redis in front of the query."
# Result: p95 = 860 ms, plus a cache to invalidate and pay for.

Better

text
# Attribute the 900 ms first. 310 ms of it is queueing for a
# database connection, which no cache in front of the query
# can shorten — the wait happens before the query runs.

What you see: A cache ships, the dashboard barely moves, and the system now has a second source of truth to invalidate and a new class of staleness bug — with the original bottleneck untouched.

Why: A cache shortens the segment it sits in front of, and nothing else. When the dominant cost is upstream of that segment — pool contention, TLS handshakes, a slow serialiser — the cache is measured against a number it was never able to change. Attribution is what stops a plausible lever from being aimed at the wrong part of the budget.

The six performance levers, what each one fixes, and when it is the wrong answer

The six performance levers, what each one fixes, and when it is the wrong answer
LeverFixesCheckWrong answer when
Right computeCPU or memory starvation, and money spent on idle capacityCPU and memory utilisation at peak, against the instance familyUtilisation is already low — the time is being spent waiting, not computing
CachingRepeated reads of data that changes slower than it is requestedHit rate, and the staleness the requirement actually allowsThe read is unique per user per request, so nothing is ever reused
Connection poolingTime spent waiting for a free connection, and database connection exhaustionPool size × worker count × task count against the engine's `max_connections`The pool is idle — the connections are being held by slow queries, not by demand
Database tuningQueries scanning far more rows than they returnThe slow query log, then an execution plan on production-shaped dataThe query is already indexed and the volume itself is the problem
CDNRound trips for anything cacheable, and origin loadCache hit ratio, and which headers and cookies the cache key includesEvery response is personalised, so the cache key is unique per request
Async processingWork the user does not need to wait forWhich steps in the handler could complete after the responseThe user genuinely needs the result before the page can render

Together

text
# The measurement that decides which lever to pull.

Before, p95 = 900 ms
  edge + network        40 ms
  waiting on a pool    310 ms   ← largest
  query execution      260 ms
  application compute  130 ms
  inline email send    160 ms

Two changes, in order of what the numbers say:
  1. pool 5 → 20 per worker, checked against max_connections
     (20 × 4 workers × 6 tasks = 480, engine limit 1000) → -290 ms
  2. move the email onto SQS, respond before it sends   → -160 ms

After, p95 = 450 ms. No cache added, no instance resized.
The two levers everybody proposes first were the two that
would have moved the least.

Remember: Measure, attribute, then choose. The six levers are compute size, cache, pool, database tuning, CDN and async — and each one targets a specific segment of the request. The cheapest improvement is usually the last one: work moved out of the request entirely, which removes time rather than shortening it.

See also: measure before scaling · the scaling toolbox · connections and database monitoring · cache keys ttl and invalidation · the cost review

The Cost Pass — Read the Bill by Driver, Not by Total

standardadvanced

A cost review looks at what the money is being spent on rather than how much it is. Sort the bill by line item, and the top three usually include something nobody designed on purpose — a NAT gateway processing traffic that could take a VPC endpoint, logs retained forever, or storage sitting in the wrong class. Those are architecture findings, not procurement findings.

Think of it as

There are two kinds of saving. Rightsizing and commitments pay less for the same architecture; lifecycle, endpoints and retention change the architecture so there is less to pay for. The second kind compounds and does not need revisiting every quarter, so it is worth finding first.

text
group the bill by usage type → separate capacity spend from accumulated architecture → remove before you shrink → leave a budget alert behind
Where an unreviewed bill actually goes

The three shaded segments are architecture choices rather than capacity choices. They are the ones a review can remove rather than shrink.

  • A stacked bar of a monthly bill divided into six segments, with the design-driven ones highlighted.
  • Compute, 38 percent: addressed by rightsizing and autoscaling.
  • NAT gateway data processing, 19 percent: addressed by S3 and DynamoDB gateway endpoints, which are free.
  • CloudWatch Logs ingestion, 14 percent: addressed by log level and retention.
  • S3 storage, 12 percent: addressed by lifecycle rules moving cold objects out of Standard.
  • Databases, 11 percent: addressed by rightsizing and commitments.
  • Cross-zone data transfer, 6 percent: addressed by topology and by keeping chatty calls inside a zone.
  • Footnote: the second, third and fourth segments total 45 percent and were never designed, only accumulated.

Reviewing the total instead of the drivers

Wrong

text
# "The bill is up 20% this quarter. Downsize everything by
#  one instance size and turn off the staging environment."

Better

text
# Group by usage type first. The 20% is NatGateway-Bytes,
# from a new service pulling model files from S3 all day.
# One gateway endpoint removes it, and nothing gets slower.

What you see: Everything gets a little slower and a little tighter, the saving is smaller than expected, and the actual driver keeps growing because nobody looked at what changed.

Why: A total tells you that something moved and nothing about what. Grouping by usage type separates the spend you chose from the spend that accumulated, and the accumulated kind is usually both larger and removable without any performance cost at all — which is the opposite trade-off from downsizing.

The cost pass — six checks, and the finding each one produces

The cost pass — six checks, and the finding each one produces
CheckWhat to look atTypical findingThe fix
RightsizingPeak CPU and memory over a representative window, per instance familyAn instance sized for a launch that never came, running at 6%Move down a size, or change family to match the actual bottleneck
AutoscalingWhether capacity follows demand, or is fixed at the peakA fixed fleet sized for the busiest hour, paid for all twenty-fourTarget tracking on the metric that actually drives load
Storage lifecycleS3 storage class distribution, EBS snapshots, old AMIsObjects written once and never read, still in Standard after two yearsA lifecycle rule per bucket prefix, plus snapshot expiry
Data transferCross-AZ traffic between your own services, and egress to the internetA chatty service pair split across zones for availability it does not needZone-aware routing where it is safe, or fewer round trips
NAT and loggingNAT gateway processed bytes; CloudWatch Logs ingested bytes per groupS3 traffic routed through NAT, and debug logging left on in productionGateway endpoints for S3 and DynamoDB; log level and retention per group
Budgets and alertsWhether a budget exists, and who receives the alertNo budget at all, so the first signal is the invoiceA budget per account with a forecast alert to a named owner

Together

bash
# Sort the bill by driver before proposing anything.

aws ce get-cost-and-usage \
  --time-period Start=2026-08-01,End=2026-09-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --query 'ResultsByTime[0].Groups | sort_by(@, &to_number(Metrics.UnblendedCost.Amount)) | reverse(@)[:8]'

# Read the usage types, not the service names. "NatGateway-Bytes"
# and "DataProcessing-Bytes" are design findings; "BoxUsage" is a
# capacity finding. They lead to completely different fixes.

Remember: Group the bill by usage type, then split the top lines into capacity you chose and architecture that accumulated. NAT data processing, log ingestion and storage class are the three that recur, all three are removable rather than merely shrinkable, and a budget alert is what stops the next one being discovered on an invoice.

See also: architecture dominates cost · aws pricing dimensions · cost tools and commitments · nat gateway cost tradeoffs · the operations review

The Operations Pass — Could Someone Else Run This at 3 a.m.?

standardadvanced

The operations pass has one governing question: could an engineer who did not build this system diagnose and recover it at three in the morning? Everything else — the dashboard, the alarms, the runbook, the rollback, the audit trail — is a component of that answer, and each one either exists as an artefact or does not.

Think of it as

Operability is a property of what is written down, not of who is available. A system operated by the person who built it looks reliable right up to the week they are on holiday. Review it as though that person cannot be reached, because the incident that matters is the one where they cannot.

text
dashboard (2 min) → alarm with an action → runbook by symptom → tested rollback → named roles → CloudTrail elsewhere
The 3 a.m. path, and where each artefact is used

Every box is a place an incident can stall. The artefact underneath is the thing that stops it stalling there.

  • A five-step incident path drawn left to right, with the artefact required at each step underneath.
  • Step 1, the page arrives: requires an alarm with a documented action, otherwise the responder does not know if it matters.
  • Step 2, orient: requires one dashboard with rate, errors, latency, saturation and dependency health, otherwise the responder hunts across consoles.
  • Step 3, diagnose: requires a runbook keyed by symptom with the exact commands, otherwise the responder reads code during the incident.
  • Step 4, act: requires a tested rollback and a documented degraded mode, otherwise the only option is to keep debugging under pressure.
  • Step 5, account for it: requires CloudTrail in a separate account, otherwise nobody can say what changed just before the incident began.
  • Footnote: the question the pass answers is whether someone who did not build the system can complete all five steps.

Confirming operability by asking the person who built it

Wrong

text
# Reviewer: "If this pages at 3 a.m., can it be recovered?"
# Author:   "Yes, you check the queue depth and restart the
#            workers."  → box ticked.

Better

text
# Give the runbook to an engineer from another team and have
# them work last month's incident from it, in daylight.
# They got to step 2 and stopped: the dashboard names the queue
# by ARN, and nothing says which service that ARN belongs to.

What you see: The system is genuinely operable by one person and by nobody else, which stays invisible until that person is unavailable — at which point recovery time is set by how long it takes to reach them.

Why: The author cannot fail the test, because the knowledge the artefacts are missing is already in their head. Only someone without that context can tell you which step is under-specified, and running it as a daylight exercise costs an hour instead of an outage.

The operations pass — six checks, each with a test you can run in daylight

The operations pass — six checks, each with a test you can run in daylight
CheckThe questionHow to test it nowFails when
DashboardsCan one screen tell you what is broken in two minutes?Open it and name the current error rate and slowest dependencyThe answer needs three consoles and a log query
AlarmsDoes every alarm have an action and an owner?List the alarms and read what each one says to doAlarms exist with no documented action, or fire weekly and are ignored
RunbooksCan someone who did not build it follow the steps?Hand it to another team and watch them work an old incidentThe runbook says "investigate the logs" instead of naming the query
Deployment safetyHow long does it take to get back to the previous version?Roll back a deployment in staging and time itRollback is theoretical, or blocked by a forward-only migration
Incident responseWho decides, who investigates, who communicates?Read the last incident record and check the roles were namedEveryone joins the call and nobody is deciding
AuditabilityCan you say who changed what, and when, without asking anyone?Pick a change from last week and find it in CloudTrailThe trail is in the same account, short-retention, or Region-limited

Together

text
# A runbook entry that passes the pass, and the one that does not.

## FAILS
Symptom:  5xx spike
Action:   Investigate the logs and restart the service if needed.

## PASSES
Symptom:  5xx rate above 2% for 5 minutes (alarm: orders-5xx)
Owner:    team-orders
Step 1:   Is it one task or all of them?
          → CloudWatch → orders-api → HTTPCode_Target_5XX by
            TargetGroup. One target = replacement, not a code bug.
Step 2:   Did anything deploy in the last hour?
          → aws deploy list-deployments --application-name orders \
              --include-only-statuses Succeeded --max-items 5
Step 3:   If a deploy correlates, roll back and confirm:
          → aws ecs update-service --cluster prod --service orders \
              --task-definition orders:PREVIOUS --force-new-deployment
          → expect 5xx below 0.5% within 4 minutes
Step 4:   If no deploy correlates, check the database first:
          → RDS → orders → DatabaseConnections and CPUUtilization
          → connections at max_connections is the usual cause
Escalate: if not recovered in 20 minutes, page team-platform.

Remember: Ask whether a stranger could recover this at 3 a.m., then check the five artefacts that decide the answer: a two-minute dashboard, alarms with actions, a runbook keyed by symptom with real commands, a rollback that has been timed, and a CloudTrail outside the account. Test it by handing the runbook to another team in daylight.

See also: runbooks and incident management · alarms dashboards and event driven actions · the fixed investigation order · answering who changed what with cloudtrail · the sustainability review

The Sustainability Pass — Provisioned Capacity Nobody Is Using

referenceadvanced

The sustainability pass looks for capacity that is provisioned and not doing work: instances running at single-digit utilisation, storage nobody reads, environments left on overnight, and jobs that recompute results that have not changed. It overlaps heavily with the cost pass, and the useful difference is that it also counts the work itself — a query that scans a hundred times more data than it returns is a finding even when the bill is small.

Four questions, and the finding each one produces

Is the capacity being used?

Utilisation at peak

single-digit CPU is a finding, not a safety margin

Unattached volumes

provisioned, billed, and serving nothing

Does capacity follow demand?

Overnight and weekends

non-production environments left running

Fixed fleet at peak size

sized for the busiest hour, paid for all of them

Is the data still needed?

Snapshots and AMIs

superseded copies with no expiry rule

Log groups

no retention set means retained forever

Is the work itself efficient?

Full reprocessing

recomputing everything to update a few rows

Scan-heavy queries

reading far more than the result contains

  • Is the capacity being used?
    • Utilisation at peak — single-digit CPU is a finding, not a safety margin
    • Unattached volumes — provisioned, billed, and serving nothing
  • Does capacity follow demand?
    • Overnight and weekends — non-production environments left running
    • Fixed fleet at peak size — sized for the busiest hour, paid for all of them
  • Is the data still needed?
    • Snapshots and AMIs — superseded copies with no expiry rule
    • Log groups — no retention set means retained forever
  • Is the work itself efficient?
    • Full reprocessing — recomputing everything to update a few rows
    • Scan-heavy queries — reading far more than the result contains

The sustainability pass — what to look for and what it usually turns out to be

The sustainability pass — what to look for and what it usually turns out to be
QuestionWhere to lookWhat it usually is
Is anything running at very low utilisation?CloudWatch CPU and memory at peak, per Auto Scaling groupA fleet sized for a launch that never arrived
Does non-production shut down?Instance running hours for dev and staging accountsEnvironments running 168 hours a week for 40 hours of use
Is storage still read?S3 storage class mix, EBS snapshot ages, unattached volumesObjects written once, never read, still in Standard
Do log groups expire?CloudWatch log group retention settingsRetention unset, which means never expires
Is any job doing more work than the result needs?Scheduled jobs, their input size, and how many rows they changeA nightly full reprocess where an incremental pass would do
What is the trade-off being made?The review notes themselvesConsolidation that removes waste and removes headroom with it

Remember: Look for provisioned capacity doing no work: low utilisation, environments running when nobody is there, storage nothing reads, and jobs recomputing what has not changed. Then write down what the efficiency costs you — usually headroom — because a pass that only records savings is not a review.

See also: efficiency levers · sustainability as a tradeoff · the cost review · the six pillars

Advertisement