beginnerintermediateadvanced

AWS · Section 19

S3 — Object Storage

Level: intermediate
Read: 30 min
Concepts: 5

S3 stores objects (data + metadata) addressed by a key inside a bucket — "folders" are only a prefix convention over a genuinely flat namespace. This section covers that core vocabulary, S3's strong read-after-write consistency and the separate guarantees of durability vs availability, multipart upload and range-request patterns for large files, the Block Public Access override that keeps buckets private by default, and how the same primitives combine into real workload shapes: static asset delivery, direct client uploads, backup/archival, and data lakes.

This section

AWS S3 Simple Storage ServicePiyush Garg

What is true here

Bucket = container, object = data + metadata, key = unique identifier — "folders" are only a shared prefix, not real directories.
S3 gives strong read-after-write consistency for object PUT/DELETE in all Regions; durability and availability are separate guarantees.
Multipart upload splits large objects into independently-retriable 5 MiB–5 GiB parts, recommended once an object nears 100 MB.
Block Public Access overrides any bucket policy/ACL that would grant public access — on by default, keep it that way absent a specific need.
The same primitives combine differently per workload: static assets + CloudFront, presigned-URL uploads, lifecycle-driven backup/archival, and data lakes queried in place.

What you will be able to do

Explain why an S3 "folder" is a prefix convention, not a real directory, and what that implies for rename cost
Distinguish strong object-data consistency from durability and availability as three separate guarantees
Choose multipart upload and range requests appropriately for large-file transfer
Verify a bucket's real public exposure by checking both Block Public Access and any policy/ACL together
Design the right S3-based workflow (static assets, uploads, backup, data lake) for a given requirement

From the S3 object model to real workload shapes

Bucket, object, key

no real folders

Strong consistency, durability

Multipart, range GETs

Block Public Access

Static assets, uploads, backup, data lake

S3 — Object Storage

The core object model, consistency/durability/availability, large-file transfer patterns, public access risk, and common workload shapes.

S3 Core Vocabulary

coreintermediate

A bucket is a container; an object (data + metadata) lives inside it, addressed by a unique key. There is no real folder structure — a "prefix" like photos/ is just a shared substring in keys, S3 fakes a folder view over a flat namespace. A bucket policy or IAM policy controls access; versioning keeps every past copy of an object; lifecycle rules move or expire objects automatically over time.

Think of it as

S3 is a giant flat filing cabinet where every folder label you see is actually just a shared prefix printed on file labels — there are no real subfolders, only keys that happen to share a string like "photos/" before the rest of the name.

What we're doing: See that a "folder" in the console is really just objects sharing a key prefix.

keys.shbash

aws s3api put-object --bucket my-bucket --key photos/2026/trip.jpg --body trip.jpg
aws s3api put-object --bucket my-bucket --key photos/2026/beach.jpg --body beach.jpg
aws s3 ls s3://my-bucket/photos/2026/  # looks like a folder listing

1: The key "photos/2026/trip.jpg" is one single string — there is no "photos" or "2026" object or folder actually created.
2: A second object shares the same prefix, which is exactly what makes the console/CLI able to render a folder-like view.
3: This listing works by matching keys that start with "photos/2026/" — it is a prefix query, not a real directory traversal.

Why this works: Understanding that "folders" are a prefix illusion over a flat namespace explains real S3 behavior that trips up POSIX-filesystem intuition — like why renaming a "folder" is actually a full copy-and-delete of every object under that prefix.

Treating an S3 "folder rename" as a fast, atomic operation

Wrong

text

# "We'll just rename the photos/2025/ folder to photos/2026/ — should be instant."

Better

text

# There is no rename operation — this requires copying every object to a new key
# and deleting the old ones, which costs time and money proportional to object count

What you see: A "quick folder rename" on a prefix with millions of objects takes hours and generates significant request costs, surprising anyone expecting filesystem-speed behavior.

Why: Because keys are flat strings and there is no real directory structure, changing a "folder name" means individually copying and deleting every object whose key starts with that prefix — there is no metadata-only operation that can do it instantly.

S3 vocabulary, container to controls

Bucket

the container, globally unique name

Object

data + metadata, addressed by a key

Key / prefix

the unique identifier — prefixes fake folders

Bucket policy / IAM

private by default, access explicitly granted

Remember: Bucket = container, object = data + metadata, key = unique identifier. "Folders" are a prefix illusion over a flat namespace — a folder rename is really a copy-then-delete of every object under that prefix. Private by default; verify, don't assume.

Consistency, Durability, and Availability

coreintermediate

S3 provides strong read-after-write consistency for PUT and DELETE in every Region — a read immediately following a successful write returns the new data, not a stale one. Durability (objects surviving over time) and availability (being reachable right now) are different guarantees. S3 is not a POSIX filesystem: no real directories, no atomic rename, no partial-file appends.

Think of it as

Durability asks "will this data still exist a year from now" — S3 answers with extremely high confidence via redundant storage. Availability asks "can I reach it right this second" — a separate, lower guarantee, because a momentary network or service issue can affect availability without ever touching durability.

text

PUT/DELETE on an object key: strongly consistent, all Regions
Bucket-level config changes: eventually consistent — allow time to propagate

What we're doing: See what strong read-after-write consistency actually guarantees, and what it does not.

consistency.txttext

PUT object → GET immediately after → always returns the new data (strongly consistent)
enable versioning on bucket → immediately write objects
→ recommended: wait ~15 minutes for the versioning change to fully propagate first

1: Object-level PUT/GET consistency is strong and immediate — there is no window where a read can return stale data after a successful write.
3: Bucket-level configuration changes are a separate, eventually-consistent mechanism — AWS's own guidance is to wait before relying on a just-enabled setting.

Why this works: Conflating these two consistency models is a common source of confusion — object data consistency and bucket configuration consistency are governed differently, and only one of them is instantaneous.

Building an application that relies on S3 behaving like a local filesystem

Wrong

python

# open a file, append a few bytes, close it — treating S3 like a local disk
with s3_mount.open('log.txt', 'a') as f:
    f.write(new_line)

Better

python

# S3 has no partial-write/append — read the whole object, modify, PUT the whole object back
current = s3.get_object(Bucket=bucket, Key='log.txt')['Body'].read()
s3.put_object(Bucket=bucket, Key='log.txt', Body=current + new_line)

What you see: Code written assuming POSIX file semantics (append, atomic rename, partial writes) either fails outright against the S3 API or silently does something much more expensive than intended (rewriting a whole object for a one-line append).

Why: S3's object model has no operation for modifying part of an existing object in place — every "edit" is really: read/generate the full new content, then PUT the whole object as a new version, which is a fundamentally different cost and consistency model than a local filesystem.

Three separate, independent guarantees

is it correctcan I reach it

about the pastabout right now

Durability: will the data still exist a year from now
Availability: can I reach it right this second
Consistency: does a read see the latest write, right now

Remember: S3 gives strong read-after-write consistency for object PUT/DELETE in all Regions — no eventual-consistency window for data. Durability (data survives) and availability (reachable now) are separate guarantees. S3 is not POSIX: no real directories, no atomic rename, no partial writes.

Multipart Upload and Large-File Patterns

standardintermediate

Multipart upload splits one object into independently-uploaded parts (5 MiB–5 GiB each, up to 10,000 parts, max object size 48.8 TiB) that S3 assembles afterward — recommended once an object approaches 100 MB. Range requests (GET with a Range header) fetch part of an object without downloading the whole thing.

Think of it as

Multipart upload is shipping a large piece of furniture as several separately-trackable, separately-retriable boxes instead of one enormous parcel that has to be redelivered whole if anything goes wrong partway through.

bash

aws s3api create-multipart-upload --bucket my-bucket --key large-file.zip
aws s3api upload-part --bucket my-bucket --key large-file.zip --part-number 1 --upload-id <id> --body part1

What we're doing: See why a single failed part in a multipart upload is cheap to recover from.

multipart-retry.txttext

Uploading a 10 GB file as 20 parts (500 MB each)
Part 14 fails due to a network blip
→ Only part 14 needs to be retried, not the other 19 already-uploaded parts

2: A network failure partway through only affects the specific part in flight at that moment.
3: The other 19 successfully-uploaded parts stay valid and do not need to be re-sent, which is exactly the resilience multipart upload is designed to provide.

Why this works: Splitting a large object into independently-retriable parts is what turns "a single dropped connection ruins a multi-gigabyte transfer" into "only the one affected part needs a retry," and also enables uploading multiple parts in parallel for higher throughput.

Multipart upload core specifications

Multipart upload core specifications
Item	Value
Maximum object size	48.8 TiB
Maximum parts per upload	10,000
Part size	5 MiB – 5 GiB (no minimum on the last part)

Together

bash

aws s3api create-multipart-upload --bucket my-bucket --key large-file.zip

Remember: Multipart upload: 5 MiB–5 GiB parts (no minimum on the last), up to 10,000 parts, max 48.8 TiB object — recommended once an object nears 100 MB. Each part retries independently. Range GETs fetch part of an object without a full download.

Public Access Risks and Block Public Access

standardintermediate

S3 Block Public Access is a bucket-level (or account-level) setting, on by default, that overrides any bucket policy or ACL trying to grant public access. AWS recommends keeping all four Block Public Access settings enabled unless a specific, deliberate use case requires public objects.

Think of it as

Block Public Access is a master switch that sits above bucket policies and ACLs — even a bucket policy that explicitly grants public read access is still blocked if the account or bucket has Block Public Access turned on, which is exactly the safety net it is designed to be.

bash

aws s3api get-public-access-block --bucket my-bucket
aws s3api put-public-access-block --bucket my-bucket --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

What we're doing: See why disabling Block Public Access alone does not immediately make a bucket public.

public-exposure.txttext

Block Public Access: disabled
Bucket policy: no public-read statement present
→ bucket is still private — disabling BPA only removes the override,
  it does not itself grant access

1: Disabling Block Public Access removes the safety override, but grants nothing on its own.
2: With no bucket policy or ACL statement actually granting public read, there is still no path for public access.
3: Both conditions — BPA disabled AND an explicit public-granting policy — have to be true together for real public exposure.

Why this works: Understanding Block Public Access as an override rather than the access grant itself clarifies why a security review has to check both layers — a permissive policy alone is contained by BPA, and BPA being off alone grants nothing without a policy.

Remember: Block Public Access overrides bucket policies/ACLs that would grant public access — on by default, keep it enabled unless a use case genuinely requires public objects. Real public exposure needs BOTH BPA disabled AND a policy/ACL actually granting access.

Common S3 Workload Patterns

standardintermediate

The same S3 primitives (buckets, storage classes, lifecycle rules, event notifications, presigned URLs) combine into different shapes depending on the workload: static assets fronted by CloudFront, direct client uploads via presigned URLs, scheduled backups with lifecycle transitions to cold storage, and a data lake where S3 is the durable, queryable storage layer under services like Athena or EMR.

Think of it as

S3 is the same warehouse for every one of these patterns — what changes is the loading dock (how data gets in), the shelving policy (lifecycle rules across storage classes), and who is allowed to walk in and grab something (access pattern).

bash

aws s3 presign s3://my-bucket/uploads/file.jpg --expires-in 300

What we're doing: See why a presigned URL avoids routing large upload bytes through the application backend.

upload-pattern.txttext

Client asks backend for a presigned PUT URL (short-lived, scoped to one key)
Backend returns the URL — no file bytes touch the backend at all
Client uploads directly to S3 using that URL

1: The backend only issues a scoped, time-limited credential — it never has to buffer or proxy any part of the actual file.
3: The upload traffic goes straight from client to S3, so backend bandwidth and compute are not consumed by file transfer at all.

Why this works: Routing large uploads through the application backend doubles the data transfer (client→backend, then backend→S3) and consumes backend compute/bandwidth for pure data movement — a presigned URL removes the backend from that path entirely while still keeping access scoped and time-limited.

Remember: The same S3 primitives combine differently per workload: static assets → S3 + CloudFront, uploads → presigned URLs (bypass the backend), backup/archival → lifecycle rules across storage classes, data lake → S3 queried in place by Athena/EMR.