1.Stream large files instead of loading them fully into memory
Reading an entire file into a variable before processing it means the file's full size sits in RAM at once, even if you only ever look at it one line or chunk at a time. Streaming reads and processes a file in small, fixed-size pieces instead, so memory use stays roughly constant no matter how large the file is.
Key features
- Memory use when loading fully scales with file size; memory use when streaming stays roughly constant, bounded by the chunk/buffer size, not the file size
- A process that loads a 2 GB file into a variable needs at least 2 GB of free RAM for that one request, on top of whatever else the process is doing
- An out-of-memory (OOM) kill from one oversized request can take down a whole worker process, dropping every other request it was concurrently handling, not just the offending one
- Streaming APIs (Node.js Readable/Writable streams, Python file-like iterators, HTTP chunked transfer encoding) process data as it arrives or as it is produced, without requiring the whole payload to exist in memory at once
- Streaming trades peak memory for a small amount of added complexity: you can no longer assume the whole file is available to seek around in freely — you generally process it front to back, one chunk at a time
Loading fully into memory vs streaming, for the same file
- Load fully into memory
- Whole 5 GB must fit in RAM before processing starts
- Peak memory use = file size
- A worker with 2 GB free gets OOM-killed
- Stream in chunks
- Each chunk (e.g. 64 KB) is processed and released
- Peak memory use = chunk size, not file size
- Same worker handles a 5 GB or a 5 MB file identically
Command
text
// Loading fully -- memory use = file size
data = read_entire_file("upload.csv") // 2 GB in RAM
process(data)
// Streaming -- memory use = chunk size, constant
for chunk in stream_file("upload.csv", chunk_size=64_KB):
process(chunk)Loading a file fully into memory makes peak memory scale with file size; streaming keeps peak memory bounded by chunk size instead. An oversized request that gets loaded fully can OOM-kill the whole worker process, not just fail that one request.