Infrastructure and Configuration Drift cheat sheet · AWS

1.Desired State and Actual State

Infrastructure as code describes a desired state: what should exist and how it should be configured. The account holds the actual state: what does exist right now. Drift is the difference between them, and it appears the moment anyone changes a resource outside the tool that created it.

Key features

A resource has drifted if any of its actual property values differ from the expected values, including a property or resource being deleted
A stack has drifted if one or more of its resources have drifted
CloudFormation compares expected values — from the template and its parameters — with the actual values as they currently exist
Only explicitly set property values are compared: defaults that were never written in the template are not tracked, so set a property even when setting it to its default
CloudFormation also detects drift on stack-level tags
Detecting drift on a stack does not detect drift on its nested stacks; those need their own detection operation

Where drift comes from, and how visible it is

Where drift comes from, and how visible it is
Source	Example	How you find out
A console change	Someone widens a security group during an incident	Drift detection, or the next deploy silently reverting it
A CLI or script change	A one-off `aws` command in a runbook	Same — nothing records the intent
An out-of-band scaling change	Desired count raised by hand to absorb load	The next deploy resets it, usually at the worst moment
Another service acting for you	An autoscaler or a managed add-on adjusting a value	Often permanent and expected — a candidate for an exception
An unmanaged resource	A bucket created by hand two years ago	It is not in any stack, so drift detection never mentions it
A template that omits a property	Encryption left unset and later enabled by hand	Not reported at all — unset properties are not compared

Two states, one gap, four ways to close it

The template and the account are separate systems that only agree when something makes them agree. Every route back to agreement is a deliberate act.

Signature

desired state (template) ≠ actual state (account) ⇒ drift · resolve by updating, redeploying, importing, or recording

The template is a claim; the account is the fact; drift is the gap, and it is the default outcome rather than an anomaly. Only explicitly set properties are compared, so set them even at their defaults, and resolve every drift by deciding which side was right and writing that decision back into the template.

2.CloudFormation Drift Detection and Review Workflows

CloudFormation can compare a stack against its template and tell you which resources differ. It reports a status per stack and per resource, and for each drifted property whether a value was added, removed or changed. Knowing what the statuses mean — especially `NOT_CHECKED` — is what stops the report being read as more reassuring than it is.

Key features

Stack drift statuses are `DRIFTED`, `IN_SYNC` and `NOT_CHECKED`; resource statuses add `DELETED` and `MODIFIED`
A stack with no resources that support drift detection also reports `IN_SYNC`
Property differences are typed as `ADD`, `REMOVE` or `NOT_EQUAL`
Drift detection can be run on stacks in `CREATE_COMPLETE`, `UPDATE_COMPLETE`, `UPDATE_ROLLBACK_COMPLETE` and `UPDATE_ROLLBACK_FAILED` states
Detection needs read permission on each resource type plus `cloudformation:DetectStackDrift`, `DetectStackResourceDrift` and `BatchDescribeTypeConfigurations`
CloudFormation does not perform drift detection on the `KMSKeyId` property of any resource, because a key can be referenced by multiple aliases

Every status, and what it actually tells you

Every status, and what it actually tells you
Status	Applies to	Means
`DRIFTED`	Stack, stack set, stack instance	One or more resources differ from the expected configuration
`IN_SYNC`	Stack, resource	Everything compared matched — including a stack with nothing comparable
`NOT_CHECKED`	Stack, resource	CloudFormation has not compared it; unsupported types land here
`MODIFIED`	Resource	At least one property value differs
`DELETED`	Resource	The resource no longer exists
`ADD`	Property difference	A value was added to an array or list property
`REMOVE`	Property difference	The property was removed from the current configuration
`NOT_EQUAL`	Property difference	The current value differs from the template value
`DETECTION_FAILED`	Operation	Detection failed for at least one resource; partial results are available

Known blind spots, and what to do about each

Known blind spots, and what to do about each
Blind spot	Why	Compensating control
Unsupported resource types	Not all types support drift detection	AWS Config rules, or a periodic audit of those types
Nested stacks	Detecting on a parent does not cover children	Run detection directly on each nested stack
Unset properties	Only explicitly set values are compared	Set the property in the template even at its default value
`KMSKeyId`	A key can be referenced by multiple aliases	Check key associations separately if it matters
Values the service never returns	Passwords and similar are never returned	Rotate through Secrets Manager rather than the template
Equal-but-not-identical values	1024 MB and 1 GB compare as different	Match the template's units to what the service reports
Unmanaged resources	They belong to no stack at all	Import them into a stack, or track them in Config

What a drift result can be

detect-stack-driftno differencesfounda resource MODIFIEDor DELETEDresource typeunsupporteddetection failedon a resourcerun again — it is asnapshot, not a monitorafterresolving

Never checked

start

DETECTION_IN_PROGRESS

IN_SYNC — everything compared matched

DRIFTED — at least one resource differs

NOT_CHECKED — unsupported or unexamined

DETECTION_FAILED — at least one resource failed

end

Signature

DRIFTED · IN_SYNC · NOT_CHECKED · (resource) MODIFIED, DELETED · (property) ADD, REMOVE, NOT_EQUAL

Drift detection compares explicitly set properties on supported resource types in one stack, and reports `DRIFTED`, `IN_SYNC` or `NOT_CHECKED`. Always report the `NOT_CHECKED` count alongside the others, run detection before production deploys and after incidents, and read the change set before deploying over a drifted stack.

3.Immutable and Reproducible Over Console-Only Changes

A console change is fast, invisible and unrepeatable. Immutable infrastructure takes the opposite approach: nothing is modified in place, so a change means building a new version and replacing the old one. The result is that the running system is always something you could rebuild, because you just did.

Key features

Immutable means replace rather than modify: a new image or task definition is deployed, and the old one is destroyed rather than patched
Reproducible means the definition lives in version control and produces the same result when applied again
A console change bypasses review, leaves no diff, and produces no record of intent — only CloudTrail's record of the API call
Changes made outside the tool can complicate later stack update or deletion operations, which is why AWS provides drift detection at all
Console access is still needed for reading, investigating and genuine emergencies — the constraint is on writing, not on looking
Every manual change is either promoted into code or reverted; leaving it in place is a third option that quietly becomes permanent

What each approach costs, honestly

What each approach costs, honestly
Aspect	Console / in-place change	Immutable through code
Time for one small change	Seconds	Minutes to hours, including review and deploy
Time for the same change across 40 accounts	Impossible to do consistently	The same as for one
Review	None	A pull request with an author and a diff
Rollback	Undo it by hand, if anyone remembers what it was	Deploy the previous version
Audit trail	The API call, without the reason	The commit, with the reason
Reproducibility after loss	Depends on who is still available to ask	Re-apply the definition
Incident response	Excellent — which is why the exception exists	Slower, so break-glass is a documented path

Modify in place, or replace

Mutable — change what is running

SSH in, edit the config, restart the service
Fast now, and the change exists nowhere but on that host
Instances diverge: each one carries its own history
Rebuilding produces a machine that behaves differently
The only record is CloudTrail, which shows the call and not the reason

Immutable — replace what is running

Build a new image or task definition, deploy, remove the old
Slower per change, and the change is reviewable and repeatable
Every instance of a version is identical to every other
Rebuilding produces exactly what is running now
The record is the commit, with an author, a diff and a reason

Signature

replace, do not modify · definition in version control · console read-only by default, writes via an assumed role

Replace rather than modify, and keep the definition in version control so the running system is always reproducible. Give read access to the console freely, put one step of friction in front of writes, and after every emergency change decide explicitly to promote it or revert it.

4.Documenting the Break-Glass Path

Some manual actions cannot be automated away: recovering when the pipeline itself is broken, acting when the identity provider is down, or making a change faster than a deploy allows. Break-glass is the sanctioned path for those moments — written down in advance, heavily logged, and reviewed afterwards.

Key features

The procedure has to exist before the emergency, because writing one under pressure produces the shortcut it was meant to prevent
Break-glass access is separate from daily access: a distinct role, requested deliberately, alerting on use rather than merely logging it
Every use is announced at the time — the alert is the announcement, not a report written afterwards
The procedure ends with drift detection and a promote-or-revert decision, so the manual change does not silently persist
It must work when the usual dependencies do not, including the identity provider and the deployment pipeline
Untested break-glass fails exactly when it is needed, so it belongs in the drill catalogue alongside every other recovery path

What must be written down before it is needed

What must be written down before it is needed
Item	Why it cannot be worked out later
Who may use it	Deciding under pressure means the least appropriate person is often the one available
How access is obtained	If it depends on the identity provider, it fails during an identity provider outage
What it may be used for	Without a boundary, break-glass becomes the normal path for anything urgent
Who is notified, and how	An alert nobody receives is a log entry, not a notification
What must be recorded	Details are lost within hours, and the review happens days later
How it is closed out	Otherwise the manual change stays, undocumented, and becomes a dependency
When it was last tested	An untested emergency path is an assumption, exactly like an untested backup

The break-glass procedure, in order

Before anything: is this really break-glass?

A slow pipeline is not an emergency. Break-glass is for when the normal path cannot work, not when it is inconvenient.

Assume the break-glass role

A distinct role, not a daily one. Assuming it fires an alert to a channel people watch, so the use is announced as it happens.

Announce, and say what you are about to do

One line in the incident channel before the change, not after. This is what makes the next hour reconstructable.

Make the smallest change that resolves the incident

One change, narrowly scoped, with a timestamp. Not the cleanup, not the improvement — the resolution.

Record it while it is fresh

What was changed, from what to what, why, and at what time. Memory of this decays within hours.

Detect drift and decide: promote or revert

Run drift detection as a closing step. Every manual change is either written into the template or removed — never left as-is.

Review the path itself

If break-glass was needed for something routine, the routine path has a gap. Fix that, not just the incident.

Command

text

# Break-glass is defined by four properties, all of them deliberate
SEPARATE   a distinct role, not an elevation of a daily one
LOUD       assuming it notifies people, in real time
BOUNDED    scoped to what an emergency actually needs
CLOSED     ends in promote-or-revert, never in "left as-is"

Write the emergency path down before the emergency: who may use it, how access works when the usual dependencies are down, what it is bounded to, who gets alerted, what gets recorded, and how it closes out with drift detection and a promote-or-revert decision. Test it on a schedule, like any other recovery path.