beginnerintermediateadvanced

Django · File Exports and Imports

File Exports and Imports

Where the bytes accumulate, and what happens when row 4,000 is wrong

StreamingHttpResponse has no content attributeit has streaming_content — and an iterator body, so memory stays flat
Headers go first, so a generator error truncatesno 500 is possible once the response has been returned
Past some size, the export is a rowrequested → building → ready/failed, with a signed URL at the end
A signed URL is a bearer credentialexpire it, randomise the key, and link emails to a page instead
Validate the whole file before writing any of itthe user wants every error with row numbers, not the first one

1.Formats, and streaming instead of building in memory

An `HttpResponse` builds the whole body in memory before anything is sent, so a 400 MB export needs 400 MB of process memory and the browser sees nothing until it is finished. `StreamingHttpResponse` takes an iterator instead and sends rows as they are produced. Django's documented CSV recipe pairs it with a tiny `Echo` class whose `write()` returns the value rather than storing it, which turns `csv.writer` into a generator. The format choice is separate: CSV for tabular data, JSON when structure matters, and Excel only when a person will actually open it in Excel.

Key features

`HttpResponse` builds the entire body in memory before the first byte is sent
`StreamingHttpResponse` "has no `content` attribute" — it has `streaming_content`
The documented CSV recipe uses an `Echo` class that "implement[s] just the write method"
Streaming avoids "a load balancer dropping a connection that might have otherwise timed out"
An exception inside the generator cannot produce a 500 — headers are already sent
`FileResponse(open_file, as_attachment=True, filename=…)` sets `Content-Disposition` for you

The three formats, and how each behaves under streaming

The three formats, and how each behaves under streaming
Format	Streams?	Use it when
CSV	yes — a row is a complete unit	tabular data; the default for a data export
JSON Lines (`.jsonl`)	yes — one object per line	structure matters and the file is large
JSON (one array)	awkward — it is one value	small responses, or an API rather than a file
`.xlsx`	no — the zip directory is written last	a person will genuinely open it in Excel
`.xlsx` via a job	n/a — built offline	the honest way to deliver a large spreadsheet

Which response class to return

Which response class to return
Class	Body is	Reach for it when
`HttpResponse`	a string, fully built	small and bounded — a few hundred rows
`StreamingHttpResponse`	an iterator	generated data of unknown or large size
`FileResponse`	an open file object	a file that already exists on disk or in storage
a redirect to a signed URL	nothing — the CDN serves it	large files: keep your process out of the transfer

One export of 800,000 orders, two response classes

HttpResponse

Every row is materialised before the first byte leaves
Peak memory scales with the export — and multiplies per concurrent user
The proxy sees a silent connection and may time it out
The user stares at a blank tab for the whole build
An error mid-build is at least an honest 500

StreamingHttpResponse

Each row is written and discarded
Memory stays roughly flat at any row count
Bytes flow immediately, so the connection stays visibly alive
The download starts at once
But: headers are already sent, so an error truncates instead of 500-ing

Command

python

StreamingHttpResponse((writer.writerow(r) for r in rows), content_type="text/csv")

`HttpResponse` accumulates the whole body in memory; `StreamingHttpResponse` takes an iterator and stays flat, and it also keeps the connection visibly alive so a proxy does not time it out. Django's `Echo` class — a `write()` that returns instead of buffering — is what turns `csv.writer` into a generator. Stream the queryset too, with `.iterator()`, or the laziness stops at the database. And remember the trade: headers go out first, so an exception inside the generator truncates the file rather than raising a 500. Do everything fallible before the response object exists.

2.When streaming is not enough: async exports and signed URLs

Streaming keeps memory flat, but it still holds a request open for the entire build and still cannot report an error once it has started. Past some size — or for a format like `.xlsx` that cannot stream at all — the export becomes a **job**: the request creates an export record and returns immediately, a worker builds the file into object storage, and the user is handed a **signed URL**, a time-limited link that grants access to that one object without your application serving the bytes.

Key features

A streaming export still occupies one request worker for the whole build
An error inside a streaming generator truncates the file; a job failure is visible and retryable
`.xlsx` cannot stream — its zip central directory is written last, so it must be built offline
Model the export as a row: requested → building → ready / failed
Signed URLs expire and let storage serve the bytes; your process never reads them
Randomise the object key — expiry protects the link, not a guessable path

Three ways to deliver an export, and what each one costs

Three ways to deliver an export, and what each one costs
Approach	Holds a request for	Errors are
`HttpResponse`	the whole build, at full memory	an honest 500
`StreamingHttpResponse`	the whole build, at flat memory	a silently truncated file
async job + signed URL	milliseconds	a failed row, retryable, with a reason
async job + emailed link	milliseconds	the same — plus the link is now in a mailbox

What a signed URL does and does not protect

What a signed URL does and does not protect
Property	Signed URL	Note
grants access without a login	yes	that is the point — and the risk
expires	yes, at the time you choose	minutes for a link in a page; hours for one in an email
limited to one object	yes	the signature covers the key
survives being forwarded	yes, until expiry	treat the URL itself as the credential
hides an unguessable key	no	randomise the key — do not use the export id
re-checks permissions	no	so re-check them when you issue the link

A 900 MB export, from click to download — with your process out of the transfer

Browser

Django

Queue

Worker

Object storage

1POST /exports (filters)
2authorise + validate, create Export(status="requested")the only work done inside the request
3on_commit → build_export.delay(id)
4202 Accepted {id, status}returns in milliseconds
5deliver job
6status = "building"; stream rows to a temp filea failure here is a retryable job, not a truncated download
7upload to exports/<random>.csv
8status = "ready", store key + row count
9GET /exports/<id> (poll, or follow an email link)
10re-check permission, then sign a short-lived URLownership can change between build and download
11302 → signed URL
12GET the object directlythe bytes never pass through Django

Command

python

return JsonResponse({"id": export.id}, status=202)   # not the file

Streaming fixes memory but still holds a request worker for the whole build and still cannot report a mid-file error. Past that point, make the export a job and the export a *row*: requested → building → ready/failed, so the UI has something to poll and a failure has somewhere to live. Build to a temporary file, not a buffer — worker memory is usually tighter than web memory, and a deterministic OOM turns retries into a loop. Store the object under a random key, hand out short-lived signed URLs so storage serves the bytes, and link emails to a page rather than to the URL, because a signed URL is a bearer credential.

3.Imports: validation, partial failure, and being safe to re-upload

An import is untrusted input in bulk, so the first decision is what happens when row 4,000 of 10,000 is invalid. There are only two honest answers — reject the whole file, or accept the good rows and report the bad ones — and you have to choose deliberately, because "it depends what the exception does" is how half a file gets imported. The second decision is what happens when the same file is uploaded twice. That is answered by a **stable business key** in the file plus a uniqueness rule in the database, which turns a re-upload into an update rather than a duplicate.

Key features

Validate every row before writing any row — the user wants all the errors, not the first
Reuse a `Form` per row: types, coercion, `cleaned_data` and messages come free
Choose all-or-nothing or per-row deliberately; the accidental middle is a half-applied file
Idempotency needs a business key from the data — row position is not stable
Enforce the key with a `UniqueConstraint`, then upsert against it
`bulk_create(..., update_conflicts=True, unique_fields=[...], update_fields=[...])` is the upsert

Two honest failure policies, and the one to avoid

Two honest failure policies, and the one to avoid
Policy	Right when	Implementation
all-or-nothing	the file is one indivisible change	validate all → one `atomic()` around the apply
partial, with a report	rows are independent records	validate all → apply valid rows → return the rejects
(avoid) stop at first error	never — it is not a decision	a prefix is applied and nobody knows where it stopped

Where each guarantee actually comes from

Where each guarantee actually comes from
Guarantee	Mechanism	Why not elsewhere
field types are right	a `Form` per row	hand-rolled parsing accepts ambiguous dates silently
no duplicates on re-upload	`UniqueConstraint` + upsert	a `.exists()` check races two concurrent uploads
a batch is all-or-nothing	`transaction.atomic()`	a loop without one leaves a prefix applied
memory stays flat	chunked reads + `bulk_create`	a 200 MB CSV read into a list is 200 MB resident
the same file is recognised	a stored hash of the bytes	filenames are not identity

Two passes over one file — and why nothing is written during the first

new filealreadyprocessedcollect, donot stopall validall-or-nothingpartial

Upload: products.csv, 10,000 rows

Hash the bytes

seen before? tell the user, do not re-apply

Pass 1 — validate every row

a Form per row; nothing is written

2 rows invalid

row 4,000: price "N/A" · row 7,412: sku blank

Which policy?

decided up front, not by what the exception does

All-or-nothing: reject the file

zero rows written; the user gets both errors

Pass 2 — apply the 9,998 valid rows

batched upserts inside atomic()

Upsert on (tenant, sku)

UniqueConstraint decides insert vs update

Report: 9,998 applied, 2 rejected

with row numbers and reasons

Command

python

Product.objects.bulk_create(objs, update_conflicts=True,
                            unique_fields=["tenant", "sku"], update_fields=["price"])

Validate the entire file before writing any of it, with a `Form` per row so you inherit coercion and messages — and report every error with its row number, not just the first. Choose all-or-nothing or per-row explicitly; the accidental middle leaves a prefix applied and nobody knows where. Idempotency comes from a business key in the *data* (SKU, invoice number), enforced by a `UniqueConstraint` and applied with an upsert — never a check-then-create, which races. Read in chunks and batch the writes, because an upload's size is the user's choice, not yours.

Spotted a mistake on this page? Tell us.

Read Sheet