Object Storage

Buckets, keys and versions as the data model; multipart upload and signed URLs as the mechanics; keeping large files off the app server

Bucket → keyflat namespace, no real folders
VersioningDELETE adds a marker, doesn't erase data
Multipart uploadparts 1-10,000, recommended at 100 MB+
Signed URLstime-limited access, no shared credentials

1.The object storage model: buckets, keys, metadata, versioning

Object storage holds data as whole, immutable objects — not files in a directory tree and not rows in a table. A bucket is a flat namespace; every object inside it is addressed by a key, a string that is the object's entire identity (there are no real subdirectories, only keys that share a prefix). Each object carries metadata alongside its bytes — content type, custom headers, timestamps. Versioning, once turned on for a bucket, keeps every past copy of an object instead of overwriting it on write or delete. Lifecycle rules then automate what happens to those objects and versions over time: moving them to cheaper storage tiers or expiring them, based on age or prefix, with no application code involved.

Key features

A bucket is a flat, globally-scoped namespace; a key is a single opaque string that is the full identity of an object inside it — there is no real directory tree underneath
An object = its data (the bytes) + its metadata (content type, custom key-value headers, timestamps) — metadata can be read without downloading the object body
A "folder" shown in a console or CLI is a prefix illusion: keys that share a leading substring like photos/2026/, matched by a query, not a real filesystem path
Versioning, once enabled on a bucket, cannot be fully turned back off (only suspended) — every subsequent write or delete produces a new version instead of overwriting or truly removing data
With versioning on, a DELETE does not erase the object — it inserts a delete marker that becomes the new current version; the prior version still exists and is still billed
Lifecycle rules act on age and prefix (e.g. "objects under logs/ older than 30 days") to transition storage class or expire objects/versions automatically, without application code

The four core concepts of the object storage model

The four core concepts of the object storage model
Concept	What it is	Key property
Bucket	A flat, named container for objects	Globally unique name; holds unlimited objects
Key	The unique string identifying an object within a bucket	No real folders — shared prefixes only look like directories
Metadata	Data about the object, stored alongside it	Readable without fetching the object body itself
Versioning	Keeps every write/delete as a distinct version	A DELETE inserts a marker; it does not erase prior versions

Versioned writes and a DELETE, on one key

Client

Bucket

1PUT inv-4471.pdfcreates v1 (current)
2PUT inv-4471.pdfcreates v2 (current), v1 kept
3DELETE inv-4471.pdfadds a delete marker (current) — v1, v2 still stored and billed

Command

text

bucket:      my-app-uploads
key:         invoices/2026/inv-4471.pdf
metadata:    Content-Type: application/pdf, uploaded-by: svc-billing
version id:  a1b2c3...   (only exists once versioning is enabled)

Bucket = flat container, key = full object identity (no real folders, only shared prefixes), metadata = data about the object stored alongside it. Versioning keeps every version; DELETE adds a marker rather than erasing data. Lifecycle rules are what actually reclaim storage from old versions — configure them together, not versioning alone.

2.Multipart upload and signed URLs

Multipart upload splits one large object into independently-uploaded parts (numbered 1 to 10,000) that the storage service reassembles once all parts arrive — AWS recommends it for objects 100 MB or larger. It lets parts upload in parallel, lets a failed part retry alone instead of restarting the whole object, and requires an explicit "complete" call at the end — an upload that is never completed or aborted just sits there, still billed for storage. A signed URL (also called a presigned URL) is a plain URL with a cryptographic signature embedded in it that grants time-limited permission to GET or PUT a specific object, generated using one specific user or role's credentials, without that caller ever handling or sharing those credentials.

Key features

Multipart upload parts are numbered 1 to 10,000; each part can be uploaded independently and in any order, then reassembled by a final "complete" call
AWS recommends multipart upload for objects 100 MB or larger — smaller objects are typically uploaded as a single PUT
An initiated multipart upload does not expire on its own — it must be explicitly completed or aborted, and storage for uploaded parts is billed until then
A signed URL embeds a signature computed from a specific caller's credentials plus an expiration time — anyone holding the URL gets that caller's access to that one action, without ever seeing the credentials themselves
Signed URLs are scoped to one action (e.g. GET this key, or PUT this key) and expire — the CLI/SDK allow up to 7 days, the console UI caps a generated link at 12 hours
If the credentials used to sign the URL are later revoked or lose permission before the URL expires, the URL stops working — the URL is not an independent grant, it is a time-boxed proof of that caller's access

Multipart upload vs a single PUT

Multipart upload vs a single PUT
Property	Single PUT	Multipart upload
Recommended for	Objects under ~100 MB	Objects 100 MB or larger
Parallelism	One request, one stream	Parts upload independently, in parallel
Failure recovery	Whole object restarts on failure	Only the failed part needs retrying
Cleanup if abandoned	Nothing to clean up — it just failed	Must explicitly complete or abort, or parts are billed indefinitely

Multipart upload: create, N parts, complete

uploadall arrivenevercompleted

CreateMultipartUpload

returns upload_id

3 parts

parallel, retry alone on failure

CompleteMultipartUpload

object assembled

Abandoned

billed indefinitely until aborted

Command

bash

# Generate a presigned URL valid for 1 hour (CLI: max 7 days)
aws s3 presign s3://my-bucket/reports/q3.pdf --expires-in 3600

Multipart upload: parts 1-10,000, recommended at 100 MB+, must be explicitly completed or aborted (or its parts are billed forever). Signed URL: one caller's time-limited, action-scoped access embedded in a URL — cannot be individually revoked, only left to expire or blocked by revoking the signer's own credentials.

3.Object storage instead of application servers for large files

When an application server sits between a client and a large file — accepting the upload, holding it in memory or on local disk, then writing it somewhere durable — every byte of that file consumes the app server's CPU, memory and network bandwidth, and the file vanishes if that server crashes before it's persisted elsewhere. Routing large durable files (videos, PDFs, backups, user uploads) directly to object storage instead removes the app server from that data path entirely: the client uploads straight to the object store (often via a signed URL so the app server never touches the bytes at all), and the app server's job shrinks to just recording metadata — the key, size, owner — in its own database.

Key features

An app server relaying a large file ties up a request-handling thread/connection and its memory or disk for the whole transfer, competing with normal request traffic for the same finite capacity
Object storage is built for durability and horizontal scale as a primary property, not a side effect of how one server happens to be configured — durability does not depend on any single application server staying up
A signed URL lets the client upload or download directly against the object store, so the app server can hand out that URL and stay off the data path entirely for the transfer itself
The app server's remaining job is metadata — recording the key, size, content type, owner and any business rules in its own database — which is small, fast, and unrelated to file size

Before (app server relays the file) vs after (direct-to-object-storage)

Before (app server relays the file) vs after (direct-to-object-storage)
Aspect	App server relays the file	Direct to object storage
App server load per upload	Scales with file size (memory/disk/bandwidth)	Roughly constant — only issues a signed URL and records metadata
Failure mid-transfer	App server crash can lose an in-flight file	Object store's own durability applies from the first byte it receives
Scaling large uploads	Requires scaling app server fleet capacity too	Object store scales independently of app server fleet size
What the app server stores	The file bytes themselves (at least transiently)	Only a reference — key, size, owner — in its own database

A 2 GB upload: relayed vs. direct-to-object-storage

App server relays it

Full 2 GB flows through the app server
Occupies a thread/worker for the whole transfer
A crash mid-transfer can lose the file

Direct to object storage

App server only issues a signed URL
Client uploads 2 GB straight to the store
App server records ~200 bytes of metadata

Signature

client --(signed URL)--> object storage, app server records metadata only

Large durable files should not flow through the application server if avoidable — let the client upload directly to object storage (typically via a signed URL) and reduce the app server's job to recording metadata. This turns a file-size-dependent cost on the app server into a roughly constant one, and moves durability onto a system built for it.

Spotted a mistake on this page? Tell us.

Read Sheet