beginnerintermediateadvanced

AWS · Section 13

EC2 Storage and Lifecycle

Level: intermediate
Read: 26 min
Concepts: 4

Storage attached to an EC2 instance behaves very differently depending on its type, and a running instance should be treated as disposable rather than hand-tuned. This section covers EBS volume types and incremental snapshots, why instance store data does not survive a stop, how image pipelines turn a fix into a new, versioned AMI, and why hand-tuning a running instance instead of replacing it is what causes configuration drift.

This section

Automate EC2 Snapshots with AWS Lifecycle ManagerServerOK

What is true here

gp3, io2, st1, and sc1 trade cost against IOPS and throughput — gp3 is the sensible default; snapshots are incremental after the first.
Instance store is fast but tied to the specific host — its data does not survive a stop, terminate, or hardware failure, unlike EBS.
Immutable infrastructure means a fix becomes a new AMI version, applied by replacing instances, never by SSH-ing into a running one.
Configuration drift accumulates one uncaptured manual fix at a time — treating instances as freely replaceable prevents it from building up.

What you will be able to do

Choose an EBS volume type based on a workload's actual IOPS and throughput needs
Explain why instance store data is lost across a stop, even though the instance itself can restart
Describe how an image pipeline turns a configuration change into a new, versioned AMI
Recognize configuration drift and explain why an instance too risky to replace is itself a symptom of it

Two axes: what persists, and how a fix propagates

Persistence

+Instance store: ephemeral, host-local
+EBS: network-attached, survives a stop
+Snapshots: incremental, S3-backed backups

Lifecycle discipline

−A fix becomes a new AMI version
−Instances are replaced, never hand-patched
−Uniformity is what prevents drift

EC2 Storage and Lifecycle

EBS volume types and snapshots, instance store vs EBS, image pipelines, and why hand-tuning a running instance causes drift.

EBS volume types and snapshots

coreintermediate

EBS is network-attached block storage that persists independently of the instance it's attached to. A snapshot is an incremental, point-in-time backup stored in S3 — the first snapshot copies everything, later ones copy only the blocks that changed.

Think of it as

EBS is a hard drive you can unplug from one computer and plug into another — the data survives regardless of which computer it's currently attached to. A snapshot is a photo album where every new photo only records what changed since the last one, not the whole scene again.

bash

aws ec2 create-snapshot --volume-id vol-0abc123 --description "pre-migration backup"

What we're doing: See why the second snapshot of an unchanged volume is nearly free, compared to the first.

incremental-snapshots.txttext

# First snapshot of a 100GB volume
snap-001: copies all 100GB of used blocks to S3

# Second snapshot, taken a day later, only 2GB changed
snap-002: copies only the 2GB of changed blocks — references snap-001 for the rest

2: The first snapshot has nothing to build on, so it copies every used block.
5: Only the delta since the last snapshot is actually copied — this is what makes frequent snapshots cheap in storage terms.

Why this works: Because each snapshot only stores its delta, deleting an "earlier" snapshot does not delete data still referenced by a later one — AWS tracks which blocks are still needed across the whole snapshot chain before actually freeing anything.

Assuming deleting the first snapshot in a chain frees all its storage immediately

Wrong

text

# "snap-001 is old, I'll delete it to save on storage costs"

Better

text

# Deleting snap-001 only removes blocks not referenced by any
# remaining snapshot — snap-002 still needs some of snap-001's data

What you see: Deleting the oldest snapshot in a chain does not reduce storage costs by the amount its size suggested it should.

Why: A later incremental snapshot can still reference blocks that only exist in an earlier one — AWS retains exactly the blocks still needed by any remaining snapshot, so deleting one snapshot in a chain rarely frees its full apparent size.

Snapshots are incremental — only the delta is copied

First snapshot

+No prior snapshot to build on
+Copies all 100GB of used blocks
+The expensive one

Second snapshot, next day

−Only 2GB actually changed
−Copies just that 2GB delta
−References the first for the rest

EBS volume types and what they fit

EBS volume types and what they fit
Type	Best for
gp3 (SSD)	General purpose — the default starting choice
io2 (SSD)	High-IOPS, low-latency — databases, latency-critical workloads
st1 (HDD)	Large, sequential throughput — big data, log processing
sc1 (HDD)	Infrequent access, lowest cost per GB

Together

bash

aws ec2 create-volume --availability-zone eu-west-1a --size 100 --volume-type gp3

Remember: A snapshot is incremental — only changed blocks are copied after the first — and gp3 is the sensible default; reach for io2 only once a workload has measured, sustained IOPS/latency needs gp3 cannot meet.

Instance store vs EBS

coreintermediate

Instance store is physically attached to the specific host an instance runs on — extremely fast, but its data is gone the moment the instance stops or the underlying host changes. EBS is network-attached and survives a stop, a reboot, and even reattachment to a different instance.

Think of it as

Instance store is a whiteboard bolted to the wall of a specific rented room — write on it all you like, but the moment you leave that specific room, the whiteboard (and everything on it) is gone. EBS is a notebook you carry with you between rooms.

text

Instance store: instance-attached, ephemeral  → survives reboot, NOT stop/terminate
EBS:            network-attached, persistent  → survives stop, terminate (unless configured to delete)

What we're doing: See instance store data disappear across a stop/start, while an EBS-backed root volume does not.

stop-start-behavior.txttext

# Instance store volume, data written before stopping
$ echo "important" > /mnt/instance-store/file.txt
$ aws ec2 stop-instances --instance-ids i-0abc123
$ aws ec2 start-instances --instance-ids i-0abc123
# file.txt is GONE — instance store does not survive stop

# EBS root volume, same sequence
$ echo "important" > /data/file.txt   # on an EBS-backed volume
$ aws ec2 stop-instances --instance-ids i-0abc123
$ aws ec2 start-instances --instance-ids i-0abc123
# file.txt is still there

2: Writing to instance store looks completely normal — nothing signals it will not survive a stop.
6: The same operation on EBS storage survives the exact same stop/start sequence unchanged.

Why this works: The difference is invisible at write time — both look like ordinary filesystem writes — which is exactly why assuming instance store behaves like EBS is such an easy, costly mistake to make.

Writing application data to instance store without realizing it is ephemeral

Wrong

text

# Database data directory pointed at an instance store volume,
# "because it's faster"

Better

text

# Database data on EBS (or instance store used only for a cache/scratch
# layer that can be safely rebuilt)

What you see: A routine instance stop (for a resize, a maintenance window, or an accidental stop) permanently destroys the database's data, with no separate action having been taken to delete anything.

Why: Instance store's speed advantage comes precisely from being physically local to one specific host — that same property means its data has no existence independent of that host, and a stop moves the instance to different underlying hardware entirely.

What survives what

Instance store

+Survives: reboot
+Lost on: stop, terminate, host failure
+Best for: caches, scratch space, replicated data

EBS

−Survives: stop, reboot
−Lost on: terminate (unless retained), explicit deletion
−Best for: anything that must persist

Remember: Instance store is fast but tied to the specific host — gone on stop, terminate, or hardware failure. EBS is network-attached and persists through a stop — use instance store only for data that can be safely lost or rebuilt.

AMIs, image pipelines, and immutable infrastructure

standardintermediate

Instead of configuring a running instance by hand and hoping to remember how, an image pipeline builds a fresh AMI from a defined recipe every time — so replacing an instance means launching a new one from that AMI, not patching the old one in place.

Think of it as

Baking a fresh loaf from a written recipe every time, instead of trying to keep one loaf fresh forever by patching it — a new loaf from the same recipe is always consistent; endlessly patching one loaf is not.

text

Recipe (Image Builder pipeline) → new AMI, versioned → Auto Scaling launches instances from it

What we're doing: Compare patching a running instance by hand against rebuilding it from an updated AMI.

mutable-vs-immutable.txttext

# Mutable: SSH in, patch by hand
ssh i-0abc123 "sudo yum update -y openssl"
# now this ONE instance differs from every instance not patched the same way

# Immutable: rebuild the AMI, replace instances
1. Image pipeline builds ami-v2 with the patched package baked in
2. Auto Scaling Group launch template updated to ami-v2
3. Old instances replaced by new ones launched from ami-v2

2: This single instance is now different from its siblings — nothing records that this happened or guarantees it happens consistently elsewhere.
6: Every new instance from this point forward is identical, because they all come from the same recorded, versioned AMI.

Why this works: The mutable approach leaves no durable record of what was actually done to which instance — the immutable approach makes "what does a healthy instance look like" a question answered by one artifact (the AMI), not by trusting that every instance received the same manual steps.

SSH-ing in to fix a running instance instead of updating the AMI and replacing it

Wrong

text

# Production issue → SSH in, patch the running instance directly, "just this once"

Better

text

# Update the image pipeline's recipe, build a new AMI, and let Auto
# Scaling replace instances with the fixed version

What you see: Months later, some instances behave differently from others in ways nobody can explain, because they each received a different sequence of manual fixes over time.

Why: A hand-patched instance drifts from what its AMI describes, and there is no record of exactly what changed — the next instance launched from the same AMI (by Auto Scaling, say) will not have the same fix, silently reintroducing the original problem.

Remember: Immutable infrastructure means a fix becomes a new AMI version, not an SSH session on a running instance — every instance launched afterward is identical because they all trace back to the same recorded artifact.

Why hand-tuned servers cause configuration drift

coreintermediate

Configuration drift is when running instances gradually diverge from what their launch template or AMI describes — each manual fix, each skipped update, each one-off tweak makes the fleet less uniform, until no two instances are quite the same anymore.

Think of it as

A fleet of identical rental cars that each get a different mechanic doing different small unrecorded fixes over the months — eventually no two cars drive quite the same, and nobody has a full list of what was changed on which one.

text

Pets: hand-tuned, feared to replace, drift accumulates
Cattle: uniform, replaced freely, drift cannot accumulate because nothing survives a replacement

What we're doing: See how drift accumulates invisibly across a series of individually reasonable manual fixes.

drift-accumulation.txttext

Week 1: instance-a gets a manual firewall rule tweak for a one-off debugging session
Week 3: instance-b gets a config file hand-edited to work around a bug
Week 6: instance-c is a fresh launch from the current AMI — has neither fix

# Three instances behind the same load balancer are now three different systems

1: A reasonable, well-intentioned fix at the time — but it exists only on this one instance.
2: Another reasonable fix, on a different instance, also captured nowhere else.
3: This instance has the original, undrifted configuration — none of the accumulated fixes ever applied to it.

Why this works: No single step here was unreasonable in isolation — the problem is that none of these fixes were captured anywhere durable (the AMI, the launch template, infrastructure code), so the fleet silently stopped being uniform without anyone deciding that should happen.

Treating a production instance as too risky to ever replace

Wrong

text

# "That instance has been running for two years — nobody wants to touch it."

Better

text

# Whatever makes that instance load-bearing and irreplaceable should be
# captured in the AMI/launch config — then replacing it becomes routine again

What you see: A single long-running instance becomes the one nobody wants to reboot, patch, or replace, because nobody is confident it can be recreated identically.

Why: An instance that has become too risky to replace is itself the symptom of accumulated, uncaptured drift — the fix is not avoiding replacement forever, but ensuring whatever makes it special gets captured in a form (AMI, launch template, IaC) that a fresh instance can also start from.

Pets vs cattle

Pets (hand-tuned)

+Feared to replace
+Manual fixes accumulate, uncaptured
+Drift builds silently over time

Cattle (uniform)

−Replaced freely, on a schedule
−Every fix lives in the AMI/launch config
−Nothing survives a replacement, so drift cannot accumulate

Remember: Configuration drift accumulates one uncaptured manual fix at a time — treating instances as freely replaceable "cattle" (not irreplaceable "pets") is what prevents drift from ever building up in the first place.