File Storage System Design

Level: advanced
Read: 14 min
Concepts: 1

A file-storage product is two systems that must stay separate. Object storage holds bytes addressed by key: cheap, effectively unbounded, and unable to answer any question about ownership, hierarchy or sharing. A metadata service holds everything else in a queryable, transactional store, which is what makes "list this folder", "who can read this", "restore last Tuesday's version" and "has this been scanned" ordinary queries rather than scans over a bucket. The upload flow then takes an unusual shape, because the bytes should travel straight from the client to storage while the decision to allow them stays with you: the metadata service authorises the request, writes a `pending` record, and returns a short-lived signed URL scoped to one key — with a multipart upload for large files, so a failed part is retried instead of a whole transfer. Completion is confirmed from a storage event rather than a client callback, since a client that finished uploading can still fail to call back, and a periodic sweep expires abandoned `pending` rows and aborts their multipart uploads, which would otherwise consume storage that no listing shows. Everything expensive then happens after the object exists and outside the request: virus scanning, checksum verification, thumbnail and preview generation, text extraction. That gap between stored and checked is real and variable, so the default state during it is the safe one — a file is visible but not downloadable until a scan passes. Finally, lifecycle rules age objects into colder storage classes and expire them on schedule, so cost tracks useful data rather than total history, and deletion is a metadata operation first with the object swept afterwards.

This section

System Design: Storage | File vs Block vs Objecttechwithshakul

What is true here

Object storage answers by key; every question a file product actually asks is a metadata query, so the two must be separate systems.
Signed URLs keep authorisation on your servers and bytes off them — proxying transfers through application servers wastes memory, connections and egress.
Write the pending metadata record before issuing the URL, and confirm completion from a storage event rather than a client callback.
Sweep abandoned uploads and abort their multipart sessions, or you pay for parts that appear in no listing.
Scanning and derivative generation are asynchronous, so the state during that gap must default to not-downloadable.

What you will be able to do

Split a file product into a metadata service and object storage, and say which system answers each user-facing question
Design an upload handshake using signed URLs and multipart uploads that never routes bytes through your application servers
Handle abandoned uploads and unconfirmed completions without leaving orphaned objects or rows
Sequence the post-upload pipeline so an unscanned file has a safe, user-visible state

Two systems, one product

The metadata service, the direct-upload handshake, and the asynchronous pipeline that follows it.

Metadata service, direct uploads and the post-upload pipeline

coreadvanced

A file-storage product is two systems that must not be one. Object storage holds bytes: it is cheap, effectively unbounded, and knows nothing about who owns a file or what it is called. A metadata service holds everything else — ownership, folder structure, versions, sharing rules, scan status, timestamps — in a database that can be queried, indexed and transacted. Keeping them separate is what makes "list my files", "who has access to this", and "show me last week's version" cheap queries rather than scans over a bucket. The upload flow then has an unusual shape, because you want the bytes to go straight from the client to object storage without passing through your servers, while keeping the decision about whether the upload is allowed on your servers. A signed URL does exactly that: your metadata service authorises the request, creates a pending record, and hands back a short-lived credential scoped to one object key; the client uploads directly, in parts if the file is large; and completion is confirmed either by the client calling back or by a storage event, at which point the metadata record moves from pending to available. Everything expensive happens after that, in the background — virus scanning, thumbnail and preview generation, text extraction, checksum verification — because none of it should hold up the upload, and a file that is stored but not yet scanned has a state, not a problem. Lifecycle rules then age objects into cheaper storage classes and delete them on schedule, which is the part that keeps a storage product's cost from growing with its total history rather than its useful data.

Think of it as

A warehouse and its index cards. The warehouse takes pallets and gives back a location code; it is very good at storing things and completely unable to answer "what did the accounting department put in here last March". The index cards answer that, and they can be sorted, cross-referenced and copied without touching a single pallet. Deliveries go straight to the loading dock rather than through the office — but the office issues the docket that lets the dock accept them, and nothing is considered received until the card is written. Inspection happens after the pallet is on a shelf, not while the truck waits.

python

def start_upload(user, filename, size):
    authorize(user, filename, size)        # your rules
    key = f"u/{user.id}/{uuid4()}"
    files.insert(id=..., key=key,          # pending
                 owner=user.id, state="pending")
    return storage.presigned_put(
        key, expires_in=900,               # short-lived
        max_size=size)                     # scoped

# completion arrives as a storage event, not a
# client promise
def on_object_created(key, etag, size):
    files.update(key, state="uploaded", etag=etag)
    scan_queue.put(key)                    # async

What we're doing: Trace a 4 GB upload from authorisation to availability, including the parts that go wrong.

upload-trace.txttext

POST /uploads  {name: "site-survey.zip",
                size: 4_294_967_296}

  metadata service:
    check quota, permissions, name policy
    INSERT files (state='pending', key='u/92/ab3f')
    initiate multipart upload -> upload_id
    return 860 signed part URLs (5 MB each)

  client:
    PUTs parts directly to object storage,
    in parallel, retrying individual failed
    parts -- not the whole 4 GB
    part 417 fails twice, succeeds on the third
    try; nothing else is affected

  client completes the multipart upload

  storage event 'ObjectCreated' fires
    metadata: state='uploaded', etag recorded

  background pipeline (none of it blocking):
    checksum verify        ->  ok
    virus scan             ->  clean  (28s)
    archive listing        ->  extracted
    state='available'

Failure branch: the client abandons the upload
  no completion, no storage event
  after 24h a sweep deletes 'pending' rows and
  aborts their multipart uploads, which also
  releases the storage the parts were consuming

12: Part-level retry is the reason multipart matters at this size. A single-request upload that fails at 90% has to start again; here one 5 MB part is retried and the other 859 are untouched.
19: The storage event, not the client, is what moves the record forward. A client that finished uploading and then crashed, lost connectivity, or was closed by the user would otherwise leave a completed object with a `pending` row forever.
27: This sweep is not optional housekeeping. Incomplete multipart uploads keep consuming storage that no listing shows, so a design without this job pays for bytes it cannot see and cannot explain.

Why this works: The trace shows the division of labour that makes the design work: your servers handle authorisation, records and orchestration — all small, fast, transactional operations — while four gigabytes never touch them. Everything slow is either the client's problem (uploading parts) or asynchronous (scanning), so no request in your system is ever waiting on a large file.

Proxying uploads and downloads through your application servers

Wrong

python

@app.post("/files")
def upload(request):
    data = request.body.read()      # 4 GB into a
    storage.put(key, data)          # worker's memory,
    return 201                      # holding a
                                    # connection for
                                    # the whole time

Better

python

@app.post("/uploads")
def start_upload(request):
    ...
    return {"url": storage.presigned_put(key)}
# the bytes never enter this process; the request
# is a few milliseconds and a database insert

What you see: Application servers run out of memory or connections during ordinary use, and scaling them up helps only until a few more large uploads arrive. Egress costs are double what they should be, because every downloaded byte is paid for twice — once out of storage, once out of your servers.

Why: A file upload occupies a worker for as long as the client's network takes, which is unbounded and unrelated to how much work your code is doing. Signed URLs move the transfer to a service built for it, leaving your servers doing the part they are good at: deciding whether the transfer is allowed and recording that it happened.

Bytes bypass your servers; decisions do not

Client

Metadata service

authorises, records pending, issues a signed URL

Object storage

receives the bytes directly, in parts if large

Storage event

object created — the authoritative completion signal

Background pipeline

scan, checksum, thumbnails, text extraction

State: available

Lifecycle rules

age to colder classes, expire on schedule

Which system answers which question

Which system answers which question
Question	Answered by	Why not the other one
What files are in this folder?	Metadata service	A bucket listing is a prefix scan with no hierarchy or permissions
Who can read this file?	Metadata service	Object storage has no notion of your users or sharing rules
Give me version 3 from last Tuesday	Metadata service, resolving to an object key	Versioning in the bucket has no user-facing history
Give me the bytes	Object storage, via a signed URL	Streaming bytes through your servers wastes bandwidth and blocks workers
Has this file been scanned?	Metadata service	Scan status is derived state, not a property of the bytes

The metadata record's lifecycle

The metadata record's lifecycle
State	Set when	Visible to the user?
pending	A signed URL is issued, before any bytes exist	As an in-progress upload only
uploaded	Storage confirms the object was written	Yes, but downloads are blocked
quarantined	Scanning has not finished, or found something	Yes, marked; downloads blocked
available	Scan passed and derivatives are generated	Yes, fully
deleted	The user deletes it; the object is swept later	No

Remember: Split the system in two: object storage holds bytes by key, and a metadata service holds ownership, hierarchy, versions, sharing and scan state where they can be queried and transacted. Authorise on your servers and transfer off them, using short-lived signed URLs and multipart uploads so a failed part is retried rather than a whole file. Record `pending` before issuing the URL, confirm completion from a storage event rather than a client callback, sweep abandoned uploads, run scanning and derivative generation in the background with a safe default state, and let lifecycle rules age and expire objects so cost tracks useful data.