File and Large Data Processing

What we're doing: Compare what happens to a worker process's memory when it loads a large upload fully vs streams it.

1  # Handler for POST /uploads, worker has 2 GB free RAM
2
3  # Loading fully:
4  def handle_upload_bad(request):
5      body = request.read_all()      # buffers entire body
6      rows = parse_csv(body)         # 2nd full copy in memory
7      save_rows(rows)
8      # A 3 GB upload needs 3+ GB free -- this worker does not have it
9
10 # Streaming:
11 def handle_upload_good(request):
12     for line in request.stream_lines():   # one line at a time
13         row = parse_csv_line(line)
14         save_row(row)
15     # Memory use is bounded by one line's size, regardless
16     # of whether the upload is 3 MB or 3 TB

5

request.read_all() forces the entire request body to exist in memory before line 6 can even start — nothing downstream can run until this line finishes.

6

parse_csv(body) commonly allocates a second, parsed copy alongside the raw body, so peak memory can be a multiple of the file size, not just equal to it.

12

request.stream_lines() yields one line at a time from the underlying connection — the next line is not read until this one has been processed and can be released.

Why this works: Streaming keeps memory use bounded by the size of one chunk, not the size of the whole file, because the process never needs more than one chunk resident at a time — the file's total size stops being a limiting factor for that worker's memory budget.

What we're doing: Trace a large video upload through chunked transfer, an asynchronous transcode job, and progress polling.

1  POST /videos  (multipart/chunked, 800 MB file, 50 chunks)
2    -> server acknowledges each chunk as it arrives
3    -> once all 50 chunks are received, server responds:
4       202 Accepted  { job_id: "vid-7731", status: "queued" }
5
6  # The HTTP request is now DONE. The client is not
7  # blocked waiting for transcoding, which has not
8  # started yet.
9
10 # A background worker picks up job vid-7731:
11 for i, segment in enumerate(split_into_segments(video)):
12     transcode(segment)
13     update_progress(job_id="vid-7731", done=i+1, total=50)
14
15 # Client polls (or subscribes via WebSocket):
16 GET /videos/vid-7731/status
17   -> { status: "processing", done: 31, total: 50 }
18   ... later ...
19   -> { status: "complete", done: 50, total: 50, url: "..." }

4

The 202 response happens the moment the upload finishes, not when transcoding finishes — the caller's HTTP connection is released long before the video is actually usable.

13

update_progress is an explicit call the worker makes after each segment — progress is not automatically visible, something has to record it as work completes.

17

The status endpoint reads the same job record the worker is writing to — polling it is how the client learns progress without holding a connection open for the whole transcode.

Why this works: Chunked transfer bounds the size of each upload unit, the 202 response decouples "received" from "processed" so the request does not block on transcoding, and progress tracking gives the caller a way to observe a job that, from the outside, would otherwise look identical whether it is 5% done or about to fail.

What we're doing: Compare a schema that stores the blob directly against one that stores only a reference to it.

1  -- BEFORE: blob stored directly in the database
2  CREATE TABLE documents_bad (
3    id UUID PRIMARY KEY,
4    filename TEXT,
5    file_data BYTEA  -- the actual PDF/image bytes, up to 100s of MB
6  );
7
8  -- AFTER: metadata only, blob lives in object storage
9  CREATE TABLE documents_good (
10   id UUID PRIMARY KEY,
11   filename TEXT,
12   object_key TEXT,     -- e.g. "documents/2026/inv-4471.pdf"
13   size_bytes BIGINT,
14   content_type TEXT
15 );
16 -- object_key points into the object store; the row itself
17 -- stays under 1 KB no matter how large the referenced file is

5

file_data BYTEA makes this row's size equal to the file's size — a table of 10,000 documents averaging 20 MB each is a 200 GB table before counting any other column.

12

object_key is a small string — the row is now a fixed, tiny size regardless of whether the referenced file is 10 KB or 10 GB.

17

This is what makes routine database work (backup, replication, index rebuilds) cheap and predictable again — none of it has to move blob bytes anymore.

Why this works: A database is built to make small, structured rows fast to query, index, and replicate — none of that design point applies to a multi-megabyte blob sitting in one column, so keeping the blob out of the database preserves the performance the database was chosen for in the first place.