beginnerintermediateadvanced

AWS · Section 62

Infrastructure and Configuration Drift

Level: advanced
Read: 35 min
Concepts: 4

Infrastructure as code describes a desired state; the account holds the actual state; drift is the difference, and it appears the moment anyone changes a resource outside the tool that created it. Because nothing reconciles the two automatically, drift is the default outcome rather than an anomaly, and detecting it is a deliberate activity. CloudFormation compares the expected values from the template and its parameters against what currently exists, reporting DRIFTED, IN_SYNC or NOT_CHECKED per stack and adding MODIFIED and DELETED per resource, with each property difference typed as ADD, REMOVE or NOT_EQUAL. Three limits matter more than the statuses themselves: only explicitly set properties are compared, so a property left out of the template is invisible; a stack whose resources do not support drift detection still reports IN_SYNC; and detecting drift on a parent stack does not cover its nested stacks. Reporting the NOT_CHECKED count next to the others is what stops a green tick being read as full coverage. Reducing drift is a discipline rather than a tool: replace rather than modify, keep every definition in version control, give console read access freely and put one step of friction in front of writes. And because some changes genuinely cannot wait — the pipeline itself is down, or the identity provider is — break-glass is written down in advance as a separate role that alerts on use, bounded in scope, and closed out with drift detection and an explicit promote-or-revert decision. An organisation that has not chosen its break-glass path still has one; it is usually a broadly permissioned automation role whose use looks entirely unremarkable in the logs.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

Nothing keeps the template and the account in step; drift is the default state.
Only explicitly set properties are compared — unset ones are silently untracked.
IN_SYNC is a statement about what was compared, not about the account.
Immutable replacement makes the running system reproducible by construction.
A break-glass path that is not chosen is chosen for you, badly.

What you will be able to do

Explain what drift is and why it accumulates without anyone acting badly
Read a drift report including its blind spots, and triage rather than revert reflexively
Place drift detection where it produces action rather than a list
Argue the immutable trade-off honestly, including where it is slower
Write a break-glass procedure that works when the usual dependencies do not

The gap, the detector, the discipline, and the sanctioned exception

Desired vs actual

drift is the default, not an anomaly

Detect it deliberately

DRIFTED · IN_SYNC · NOT_CHECKED

Replace, do not modify

writes go through code

Sanction the exception

written down, alerted, closed out

Infrastructure and Configuration Drift

What drift is and where it comes from, how CloudFormation detects it and what it cannot see, the discipline that prevents most of it, and the documented exception for when none of that can apply.

Desired State and Actual State

coreintermediate

Infrastructure as code describes a desired state: what should exist and how it should be configured. The account holds the actual state: what does exist right now. Drift is the difference between them, and it appears the moment anyone changes a resource outside the tool that created it.

Think of it as

The template is a claim about reality, not a copy of it. Nothing keeps the two in step automatically — a template is only applied when someone runs it, and the account accepts changes from anyone with permission. Drift is therefore the default outcome, and detecting it is a deliberate activity.

What we're doing: Decide what to do about a drifted resource, rather than reflexively reverting it.

triage-drift.txttext

Drift report: sg-app has an extra inbound rule, 0.0.0.0/0 on 443,
added four days ago. The template does not contain it.

The reflex is "revert it". That is right about half the time, and
the other half it causes an incident. Ask three questions first.

1. WHO CHANGED IT, AND WHY?
   CloudTrail names the principal and the time. Four days ago at
   16:40, during the incident recorded in the same window. It was
   deliberate, under pressure, and it may still be load-bearing.

2. IS IT STILL NEEDED?
   Ask the person. If it was a temporary mitigation that was never
   removed, reverting is correct and overdue. If it is now how a
   partner reaches the service, reverting is an outage.

3. WHERE SHOULD THE ANSWER LIVE?
   Whichever way it goes, the outcome belongs in the template:
     still needed  → add the rule to the template, with a comment
                     and a link to the decision
     not needed    → redeploy to remove it, and note the incident
                     in the change record

The failure mode is not choosing wrongly. It is choosing silently
and leaving the template and reality still disagreeing.

1: A drift report is a question, not an instruction — the tool cannot know whether the change was right.
7: CloudTrail turns an anonymous difference into an attributable decision with a time and a person.
13: Reverting a load-bearing manual change is one of the more common self-inflicted incidents.
19: Either resolution ends with the template matching reality; that is what makes the next drift report meaningful.

Why this works: Drift is information about how the system is really operated. Treating every difference as a violation to be reverted destroys that information and creates outages; treating none of them as violations lets the template quietly become fiction. The middle path — attribute, decide, then write the decision into the template — is what keeps drift detection useful over time.

Treating a passing drift check as proof the template is complete

Wrong

text

# "Stack status IN_SYNC — infrastructure matches the code."

Better

text

# IN_SYNC means: every explicitly set property, on every
# supported resource type, in this stack (not its nested ones),
# matches. Everything else is simply not being checked.

What you see: A resource type that does not support drift detection, or a property never written in the template, changes without anyone learning about it — while the dashboard stays green.

Why: IN_SYNC is a statement about the comparison that was performed, not about the account. Resources that do not support drift detection are reported as NOT_CHECKED, unset properties are excluded by design, and nested stacks are not covered — so the green status is bounded in three directions at once.

Two states, one gap, four ways to close it

The template and the account are separate systems that only agree when something makes them agree. Every route back to agreement is a deliberate act.

Where drift comes from, and how visible it is

Where drift comes from, and how visible it is
Source	Example	How you find out
A console change	Someone widens a security group during an incident	Drift detection, or the next deploy silently reverting it
A CLI or script change	A one-off `aws` command in a runbook	Same — nothing records the intent
An out-of-band scaling change	Desired count raised by hand to absorb load	The next deploy resets it, usually at the worst moment
Another service acting for you	An autoscaler or a managed add-on adjusting a value	Often permanent and expected — a candidate for an exception
An unmanaged resource	A bucket created by hand two years ago	It is not in any stack, so drift detection never mentions it
A template that omits a property	Encryption left unset and later enabled by hand	Not reported at all — unset properties are not compared

Together

text

# The property that is invisible to drift detection
Resources:
  DataBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: acme-orders-prod
      # VersioningConfiguration is not set here.

# Someone enables versioning by hand. CloudFormation reports
# IN_SYNC, because it only compares properties the template
# explicitly sets. Set the property — even to its default — to
# make it tracked.

Remember: The template is a claim; the account is the fact; drift is the gap, and it is the default outcome rather than an anomaly. Only explicitly set properties are compared, so set them even at their defaults, and resolve every drift by deciding which side was right and writing that decision back into the template.

CloudFormation Drift Detection and Review Workflows

coreadvanced

CloudFormation can compare a stack against its template and tell you which resources differ. It reports a status per stack and per resource, and for each drifted property whether a value was added, removed or changed. Knowing what the statuses mean — especially `NOT_CHECKED` — is what stops the report being read as more reassuring than it is.

Think of it as

Read a drift report as three buckets, not two: matched, differed, and not examined. The third bucket is the one that misleads, because a resource type that does not support drift detection produces the same green-looking stack as one that genuinely matches.

What we're doing: Put drift detection into a workflow so someone actually acts on it.

drift-workflow.txttext

Running detection by hand once a quarter produces a long list
nobody triages. Three placements produce action instead.

1. BEFORE EVERY PRODUCTION DEPLOY
   Detect drift, then create a change set. If the stack has
   drifted, the change set is being computed against a template
   that no longer describes reality — and the deploy may quietly
   revert someone's deliberate change.

   Gate the pipeline: drifted → a human decides before it runs.

2. ON A SCHEDULE, WITH THE RESULT ROUTED TO AN OWNER
   An EventBridge schedule triggers detection; results go to the
   team named in the Owner tag, not to a shared channel. Include
   the NOT_CHECKED count in the message so the blind spot is
   visible alongside the findings.

3. AFTER EVERY INCIDENT
   Incidents are where deliberate manual changes are made under
   pressure, and where they are most often forgotten. Detection
   as a closing step of the incident review turns "we changed
   something at 2am" into a specific list.

The change-set habit is the highest value of the three: it makes
the deploy show what it is about to do, against reality rather
than against the last known template.

1: Placement is the whole problem — the tool works, and the report is usually ignored because it arrives with no decision attached.
7: This is the case where drift causes an outage rather than merely representing untidiness.
14: Routing by Owner tag is where the tagging schema earns its keep in an unrelated workflow.
20: Post-incident is when the highest-value drift exists and when it is most likely to be forgotten.

Why this works: Drift detection is a snapshot, not a monitor: it tells you the truth at the moment you run it and nothing after. Putting it at the three moments where drift is either most likely or most dangerous turns an occasional audit into a control, and pairing it with change sets means a deploy shows its actual effect rather than its intended one.

Deploying over drift without reading the change set

Wrong

text

# Stack is DRIFTED. Deploy anyway: "the template is the truth."

Better

text

# Create a change set and read it. It shows what the deploy will
# do to the drifted resources — which may include reverting a
# deliberate fix that is currently keeping production up.

What you see: A routine deploy removes an emergency change from three days ago and reintroduces the incident it was mitigating.

Why: A deploy applies the template regardless of why reality differs, so drift plus deploy equals silent revert. The change set is the only place that difference is visible before it is applied, which makes reading it the cheapest possible safeguard.

What a drift result can be

Never checked

start

DETECTION_IN_PROGRESS

IN_SYNC — everything compared matched

DRIFTED — at least one resource differs

NOT_CHECKED — unsupported or unexamined

DETECTION_FAILED — at least one resource failed

end

Every status, and what it actually tells you

Every status, and what it actually tells you
Status	Applies to	Means
`DRIFTED`	Stack, stack set, stack instance	One or more resources differ from the expected configuration
`IN_SYNC`	Stack, resource	Everything compared matched — including a stack with nothing comparable
`NOT_CHECKED`	Stack, resource	CloudFormation has not compared it; unsupported types land here
`MODIFIED`	Resource	At least one property value differs
`DELETED`	Resource	The resource no longer exists
`ADD`	Property difference	A value was added to an array or list property
`REMOVE`	Property difference	The property was removed from the current configuration
`NOT_EQUAL`	Property difference	The current value differs from the template value
`DETECTION_FAILED`	Operation	Detection failed for at least one resource; partial results are available

Together

text

# Run it, then read the per-resource detail — the stack status
# alone hides which resource and which property
aws cloudformation detect-stack-drift --stack-name orders-prod
aws cloudformation describe-stack-resource-drifts \
  --stack-name orders-prod \
  --stack-resource-drift-status-filters MODIFIED DELETED

# Count NOT_CHECKED separately. A stack that is 60% NOT_CHECKED
# and 40% IN_SYNC still reports IN_SYNC.

Known blind spots, and what to do about each

Known blind spots, and what to do about each
Blind spot	Why	Compensating control
Unsupported resource types	Not all types support drift detection	AWS Config rules, or a periodic audit of those types
Nested stacks	Detecting on a parent does not cover children	Run detection directly on each nested stack
Unset properties	Only explicitly set values are compared	Set the property in the template even at its default value
`KMSKeyId`	A key can be referenced by multiple aliases	Check key associations separately if it matters
Values the service never returns	Passwords and similar are never returned	Rotate through Secrets Manager rather than the template
Equal-but-not-identical values	1024 MB and 1 GB compare as different	Match the template's units to what the service reports
Unmanaged resources	They belong to no stack at all	Import them into a stack, or track them in Config

Together

text

# The false positive AWS documents, and its fix
expected: 1 GB        actual: 1024 MB      → reported as drift

# Equal, not identical. Change the template to the units the
# service reports and the false positive disappears — which
# matters because a report full of false positives stops
# being read at all.

Remember: Drift detection compares explicitly set properties on supported resource types in one stack, and reports `DRIFTED`, `IN_SYNC` or `NOT_CHECKED`. Always report the `NOT_CHECKED` count alongside the others, run detection before production deploys and after incidents, and read the change set before deploying over a drifted stack.

Immutable and Reproducible Over Console-Only Changes

coreadvanced

A console change is fast, invisible and unrepeatable. Immutable infrastructure takes the opposite approach: nothing is modified in place, so a change means building a new version and replacing the old one. The result is that the running system is always something you could rebuild, because you just did.

Think of it as

Ask of any running resource: if this vanished right now, could I recreate it exactly, from something in version control, without asking anyone what was done to it? Every console change moves the answer from yes to no, and the answer is only checked on the day it matters.

What we're doing: Make console changes hard enough to be deliberate, without blocking investigation.

read-mostly-console.txttext

The goal is not "nobody uses the console". It is "nobody
changes production in the console without meaning to".

WHAT PEOPLE GET BY DEFAULT
  Read access to everything in production: consoles, logs,
  metrics, traces, configuration. Investigation must never
  require an approval step, because that is where the tool gets
  worked around.

WHAT REQUIRES ASSUMING A DIFFERENT ROLE
  Write access to production resources. It exists, it works, and
  assuming it is logged and visible. The friction is one command,
  which is enough to make it a decision rather than a reflex.

WHAT THE PIPELINE DOES
  Everything routine. If a change is made regularly, it belongs
  in code, and the pipeline is faster than the console for it.

WHAT HAPPENS AFTER A BREAK-GLASS CHANGE
  Drift detection runs as part of closing the incident. Every
  manual change is then either promoted into the template or
  reverted — with a name against the decision. Leaving it is not
  one of the two options, because that is how a temporary fix
  becomes an undocumented permanent dependency.

1: Framing matters: a policy that reads as "the console is forbidden" gets circumvented, and one that reads as "writes are deliberate" does not.
5: Read access without friction is what makes the write friction acceptable to the people affected by it.
12: One command of friction is enough — the aim is to interrupt the reflex, not to prevent the action.
18: This step is the one that actually keeps the template true, and it is the one most often skipped.

Why this works: Immutable infrastructure fails in practice not because people disagree with it but because the emergency path is undefined, so the console becomes the emergency path and then stays the ordinary one. Separating read from write, keeping read frictionless, and forcing a promote-or-revert decision after every manual change makes the discipline survive real incidents.

Fixing an instance instead of replacing it

Wrong

text

# Instance failing health checks. SSH in, restart the agent,
# tweak a limit, mark it healthy again. Ticket closed.

Better

text

# Terminate it. The Auto Scaling group replaces it with a fresh
# instance built from the current image.
# If it fails again, the image or the configuration is wrong —
# fix that, and the fix applies to every instance.

What you see: The same fault recurs on different instances, each is fixed differently, and no two machines in the fleet are the same after six months.

Why: Repairing one instance treats the symptom on one machine and leaves the cause in the image that every other instance was built from. Replacement forces the fix to happen where it applies to everything, and keeps the fleet identical — which is what makes any later diagnosis meaningful.

Modify in place, or replace

Mutable — change what is running

+SSH in, edit the config, restart the service
+Fast now, and the change exists nowhere but on that host
+Instances diverge: each one carries its own history
+Rebuilding produces a machine that behaves differently
+The only record is CloudTrail, which shows the call and not the reason

Immutable — replace what is running

−Build a new image or task definition, deploy, remove the old
−Slower per change, and the change is reviewable and repeatable
−Every instance of a version is identical to every other
−Rebuilding produces exactly what is running now
−The record is the commit, with an author, a diff and a reason

What each approach costs, honestly

What each approach costs, honestly
Aspect	Console / in-place change	Immutable through code
Time for one small change	Seconds	Minutes to hours, including review and deploy
Time for the same change across 40 accounts	Impossible to do consistently	The same as for one
Review	None	A pull request with an author and a diff
Rollback	Undo it by hand, if anyone remembers what it was	Deploy the previous version
Audit trail	The API call, without the reason	The commit, with the reason
Reproducibility after loss	Depends on who is still available to ask	Re-apply the definition
Incident response	Excellent — which is why the exception exists	Slower, so break-glass is a documented path

Together

text

# The honest trade, stated as it actually is
# Immutable is SLOWER for the first change and FASTER for the
# hundredth, the one made across many accounts, the one that
# must be undone, and the one someone else has to understand.
#
# Teams that adopt it for the first change find it burdensome.
# Teams that adopt it for the hundredth find it obvious.

Remember: Replace rather than modify, and keep the definition in version control so the running system is always reproducible. Give read access to the console freely, put one step of friction in front of writes, and after every emergency change decide explicitly to promote it or revert it.

Documenting the Break-Glass Path

standardadvanced

Some manual actions cannot be automated away: recovering when the pipeline itself is broken, acting when the identity provider is down, or making a change faster than a deploy allows. Break-glass is the sanctioned path for those moments — written down in advance, heavily logged, and reviewed afterwards.

Think of it as

Break-glass is not an exception to the controls; it is one of the controls. Its job is to make the emergency path the one people actually take, so the alternative — a shared credential, an over-privileged role, an undocumented action — never becomes the habit.

text

# Break-glass is defined by four properties, all of them deliberate
SEPARATE   a distinct role, not an elevation of a daily one
LOUD       assuming it notifies people, in real time
BOUNDED    scoped to what an emergency actually needs
CLOSED     ends in promote-or-revert, never in "left as-is"

What we're doing: Show what a break-glass path costs when it does not exist.

no-break-glass.txttext

02:14  Deployment pipeline is down. Production needs a config
       change to stop a cascading failure.

02:16  The documented process is "deploy through the pipeline".
       There is no other documented process.

02:19  Someone uses a service role that happens to have broad
       permissions, from a laptop, because it is what works.
       No alert fires — it is a role used by automation all day,
       so its use looks entirely normal.

02:31  Incident resolved. The change is not recorded anywhere
       except CloudTrail, and nobody looks at CloudTrail.

Three weeks later
       A deploy reverts the change. The cascading failure returns.
       Nobody connects the two, because the original change was
       never written down and the person who made it has moved
       to another team.

What break-glass would have changed: the role is distinct, so the
use alerts; the alert is the announcement; the close-out step
forces promote-or-revert; and the change is in the template
before the next deploy touches it.

1: The pipeline being the broken component is the scenario break-glass exists for, and the one most procedures forget.
6: A role used by automation is the worst possible emergency credential precisely because its use is unremarkable.
12: The three-week gap is the real cost: the incident recurs and the connection is unfindable.

Why this works: Every organisation has a break-glass path; the only question is whether it was chosen. An unchosen one uses whichever credential happens to work, produces no signal, and leaves changes that outlive everyone's memory of them. Writing it down costs an afternoon and converts all three of those into a logged, alerted, closed-out action.

The break-glass procedure, in order

Before anything: is this really break-glass?

A slow pipeline is not an emergency. Break-glass is for when the normal path cannot work, not when it is inconvenient.

Assume the break-glass role

A distinct role, not a daily one. Assuming it fires an alert to a channel people watch, so the use is announced as it happens.

Announce, and say what you are about to do

One line in the incident channel before the change, not after. This is what makes the next hour reconstructable.

Make the smallest change that resolves the incident

One change, narrowly scoped, with a timestamp. Not the cleanup, not the improvement — the resolution.

Record it while it is fresh

What was changed, from what to what, why, and at what time. Memory of this decays within hours.

Detect drift and decide: promote or revert

Run drift detection as a closing step. Every manual change is either written into the template or removed — never left as-is.

Review the path itself

If break-glass was needed for something routine, the routine path has a gap. Fix that, not just the incident.

What must be written down before it is needed

What must be written down before it is needed
Item	Why it cannot be worked out later
Who may use it	Deciding under pressure means the least appropriate person is often the one available
How access is obtained	If it depends on the identity provider, it fails during an identity provider outage
What it may be used for	Without a boundary, break-glass becomes the normal path for anything urgent
Who is notified, and how	An alert nobody receives is a log entry, not a notification
What must be recorded	Details are lost within hours, and the review happens days later
How it is closed out	Otherwise the manual change stays, undocumented, and becomes a dependency
When it was last tested	An untested emergency path is an assumption, exactly like an untested backup

Together

text

# The one-page card, kept somewhere reachable when AWS is not
WHEN      the pipeline cannot deploy, or an outage needs a change
          faster than a deploy allows
WHO       any on-call engineer for this service
HOW       assume role BreakGlassProd in account 777788889999
          (MFA required; assuming it alerts #sec-alerts)
BOUND     one change, in this service. Not a cleanup, not a
          config improvement.
RECORD    in the incident channel, at the time, in one line
CLOSE     drift detection → promote to the template or revert,
          with a name against the decision
TESTED    2026-08-19, quarterly

Remember: Write the emergency path down before the emergency: who may use it, how access works when the usual dependencies are down, what it is bounded to, who gets alerted, what gets recorded, and how it closes out with drift detection and a promote-or-revert decision. Test it on a schedule, like any other recovery path.