beginnerintermediateadvanced

Django · Large Data Operations

Large Data Operations

Ten rows and ten million rows are different problems — bulk writes, in-place updates, and resumable jobs

Bulk writes skip save() and every signalthat is where the speed comes from, and where the data-quality bugs come from
If the value needs no Python, the rows should not moveupdate() with F() is one statement, and it closes the read-modify-write race
Put the transaction INSIDE the delete loopoutside it keeps every drawback of one huge DELETE
Migrations should be schema-proportionaldata-proportional work belongs in a job that can be watched and resumed
Resume from the last primary key, not an offsetan offset shifts when rows change — the same reason cursors beat OFFSET

1.`bulk_create()`, `bulk_update()`, and what they skip

Calling `save()` in a loop over 50,000 objects issues 50,000 statements. `bulk_create()` and `bulk_update()` collapse that into a handful, and `batch_size` controls how many rows go in each one. The price is that they are deliberately dumb: Django's documentation states that "the model's `save()` method will not be called, and the `pre_save` and `post_save` signals will not be sent". Anything your `save()` override does — stamping a field, updating a search vector, writing an audit row — silently does not happen. That is a feature when you want raw speed and a serious bug when you did not know about it.

Key features

Both skip `save()` and both skip `pre_save`/`post_save` — quoted directly from the QuerySet reference
`bulk_create` "does not work with child models in a multi-table inheritance scenario" and does not handle many-to-many relations
Primary keys are set after `bulk_create` on PostgreSQL, MariaDB and SQLite 3.35+ — but `ignore_conflicts` "disables setting the primary key on each model instance"
`update_conflicts=True` with `update_fields` and `unique_fields` gives an upsert rather than an error
`bulk_update()` returns the number of objects updated and cannot update the primary key
`bulk_update()` "prepares all of the WHEN clauses for every object across all batches before executing any queries"

Which write to reach for

Which write to reach for
Goal	Use	Loses
insert many new rows	`bulk_create(objs, batch_size=…)`	`save()`, signals, m2m, MTI children
insert, ignoring duplicates	`bulk_create(..., ignore_conflicts=True)`	the above and returned primary keys
insert or update on a key	`bulk_create(..., update_conflicts=True, update_fields=…, unique_fields=…)`	`save()`, signals
change loaded objects	`bulk_update(objs, ["field"], batch_size=…)`	`save()`, signals; cannot touch the pk
set a value for every matching row	`qs.update(field=…)`	`save()`, signals — and never loads the rows

Every argument is a decision — and one of them is silent

Order.objects.bulk_create(new_orders, batch_size=500, update_conflicts=True, update_fields=["total"], unique_fields=["reference"])

bulk_create

skips save() and signals — The documented caveat: "the model's save() method will not be called, and the pre_save and post_save signals will not be sent." Read the model's save() before choosing this.

new_orders

unsaved instances, built in memory — The list itself is a memory cost. For very large inputs, build and write it in chunks rather than assembling millions of objects first.

batch_size=500

rows per statement — Bounds statement size so the database or driver does not reject it. Without it Django uses one statement for everything, which can be very large.

update_conflicts=True

upsert instead of error — A conflicting row is updated rather than raising `IntegrityError`. This is what makes a re-run of an import safe.

update_fields=["total"]

what a conflict overwrites — Only these columns are updated on conflict. Omitting a field means an existing row keeps its current value for it — which is usually what you want for `created_at`.

unique_fields=["reference"]

what counts as a conflict — Must match a real unique constraint on the table. This is the column the database uses to decide "the same row", so it defines what a re-run means.

Command

python

Order.objects.bulk_create(objs, batch_size=500)
Order.objects.bulk_update(objs, ["total", "state"], batch_size=500)

Bulk writes write rows; they do not save objects. `save()`, `pre_save` and `post_save` are all skipped, which is where the speed comes from and where the data-quality bugs come from — so read the model's `save()` and its signal receivers first, then reproduce that work in the loop that builds the objects, in bulk. Use `update_conflicts` with `unique_fields` to make an import re-runnable, and keep `created_at` out of `update_fields`. Bound both sides: `.iterator()` on the read, a flush counter on the write, because `bulk_update` builds every `WHEN` clause before it executes anything.

2.Database-side updates, and deleting in batches

The cheapest way to change a million rows is not to load them. `qs.update(...)` is "performed at the SQL level" — one statement, no rows fetched, no Python objects built — and `F()` lets the new value be computed from the old one inside the database. The same idea does not extend to deletes, though: one `DELETE` over a million rows takes locks for the whole operation and produces a very large transaction, so deletes are done in bounded batches instead. Both skip `save()` and signals, exactly like the bulk methods.

Key features

`update()` is "performed at the SQL level", "returns the number of rows matched", and does not call `save()` or send signals
`F("views") + 1` is evaluated by the database, so it is safe against concurrent writers in a way read-modify-write is not
`update()` cannot be called on a sliced queryset — bound it with a `pk__in` subquery instead
A single huge `DELETE` holds locks for the whole statement and creates one very large transaction
Batched deletes must be *outside* one big `atomic()` — one transaction per batch is the point
`qs.delete()` may fan out: Django collects related rows to apply `on_delete`, so cascades do work you did not write

Where the work should happen

Where the work should happen
Change	Do it	Cost
set a constant	`qs.update(state="archived")`	one statement, no rows loaded
derive from the old value	`qs.update(views=F("views") + 1)`	one statement, concurrency-safe
derive from another column	`qs.update(total=F("net") + F("tax"))`	one statement
needs Python or an API call	`.iterator()` + `bulk_update()`	chunked read, batched write
delete a large range	loop of `pk__in` batches	many small transactions, interruptible

One million rows, two ways to change them

Load, change, save

Every row travels to the application and back
A model instance built for each one
One `UPDATE` per row
Read-modify-write: two concurrent runs lose one increment
Cost grows with the row count in three dimensions at once

Change in place

No rows fetched, no instances built
One statement, whatever the row count
The database computes the new value from the old
Concurrency-safe: the increment happens where the row is locked
Skips `save()` and signals — deliberately

Command

python

qs.update(field=F("field") + 1)          # computed in the database
qs.filter(pk__in=qs.values("pk")[:5000]).delete()   # one bounded batch

If the new value does not need Python, the rows should never leave the database — `update()` with `F()` is one statement whose cost does not grow the way a loop does, and it closes the read-modify-write race as a side effect. Deletes are the exception: batch them, and put the transaction *inside* the loop so each batch commits, releases its locks, and leaves the job resumable. Delete children explicitly rather than trusting a cascade to stay small, and pause between batches so replicas and vacuum keep up. Both paths skip `save()` and signals, exactly like the bulk methods.

3.Data migrations, background workers, and resumable work

A `RunPython` migration runs inside the deploy, and the deploy has a timeout. Backfilling ten million rows there blocks every deploy behind it and, if it fails partway, leaves you with a migration recorded as unapplied over a table that is already half-changed. The rule that follows: a migration should make the *schema* ready and hand large data work to a background job that can be run, watched, interrupted and resumed. When the work must live in the migration, make it chunked and idempotent, and mark it `elidable=True` so it can be squashed away later.

Key features

`RunPython` gets `apps` — use `apps.get_model()` for the *historical* model, never the imported class
Migrations run inside the deploy; a long backfill blocks every deploy behind it
On PostgreSQL each migration runs in a transaction by default, so a huge backfill is one huge transaction
`atomic = False` on the Migration class lets a long data migration commit in pieces
`elidable=True` marks a `RunPython` as safe to drop when migrations are squashed
Resume from the last processed primary key, not an offset — an offset moves when rows change

Where the work belongs

Where the work belongs
Work	Migration?	Why
add a nullable column	yes	fast, schema-proportional
backfill 10,000 rows	yes, chunked + `elidable`	short enough to sit inside a deploy
backfill 10,000,000 rows	no — a job	duration is data-proportional; the deploy would time out
recompute a derived field nightly	no — a scheduled task	recurring by nature
a one-off export of everything	no — a job	needs progress, retries, and an artifact

A long backfill, and the two paths out of a failure

batchcommitsnext batchno rows leftanythinggoes wrongbatched +idempotentone big transactioninstead

queued

start

running — batch n

batch committed, marker saved

interrupted (deploy, OOM, timeout)

resumed from the last marker

complete

end

all progress rolled back

end

Command

python

class Migration(migrations.Migration):
    atomic = False        # so a chunked backfill can commit in pieces
    operations = [migrations.RunPython(backfill, migrations.RunPython.noop, elidable=True)]

Keep migrations schema-proportional and hand data-proportional work to a job — a migration runs in the deploy path, has a deploy timeout, and its unit of recovery is all-or-nothing. When a backfill must be a migration, use `apps.get_model()` for the historical model, filter on what is *not yet done* so a re-run finishes the remainder, set `atomic = False` so batches commit, and mark it `elidable=True`. For any long job, resume from the last primary key rather than an offset, and record that marker durably.

Spotted a mistake on this page? Tell us.

Read Sheet