beginnerintermediateadvanced

Django · Section 74

Large Data Operations

Level: advanced
Read: 34 min
Concepts: 3

The roadmap puts it plainly: ten rows and ten million rows call for different strategies. The first shift is from saving objects to writing rows. `bulk_create()` and `bulk_update()` collapse thousands of statements into a handful, and Django states the cost without ambiguity — the model's `save()` is not called and `pre_save`/`post_save` are not sent. Every convenience an application layers on top of saving lives in exactly those two places: derived fields, denormalised counters, search vectors, audit rows, cache invalidation. So the question before every bulk write is what else used to happen, answered by reading the model rather than by assuming, and the fix when something did matter is to do that work in bulk as well. `batch_size` is the other half, and it matters for a reason beyond statement size: `bulk_update` prepares the `WHEN` clauses for every object across all batches before it executes anything, so a single call with a huge list is a memory cost regardless. The second shift is to stop moving rows at all. When the new value does not depend on Python, `update()` with `F()` expresses it as one SQL statement — no rows fetched, no instances built — and it closes the read-modify-write race that loses increments under load, because the read and the write become the same statement. Deletion is the exception that proves the rule: one enormous `DELETE` holds locks for its whole duration and creates a transaction big enough to hurt replication, so it is done as a loop of bounded batches with the transaction *inside* the loop. Putting the transaction outside keeps every drawback and adds none of the benefits, which is the most common way this pattern is written wrong. The third shift is about where long work lives. A migration runs in the deploy path, has a deploy timeout, and succeeds or fails as a whole — none of which suits a backfill whose duration is proportional to data volume. Keep migrations schema-proportional and give large backfills to a job that can be watched, throttled, interrupted and resumed. When a backfill must be a migration, use the historical model from `apps.get_model()`, filter on what is not yet done so a restart finishes only the remainder, let batches commit with `atomic = False`, and mark it `elidable=True`.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

Bulk writes skip save() and both save signals — check what that loses before using them.
batch_size bounds statement size, but bulk_update still builds all WHEN clauses up front.
update() + F() changes rows without loading them and removes the read-modify-write race.
Batch deletes with the transaction inside the loop, so each batch commits and releases its locks.
Long backfills belong in resumable jobs; migrations should stay schema-proportional.

What you will be able to do

Replace a save() loop with a bulk write without silently dropping derived fields
Write an import that is safe to run twice
Purge millions of rows from a live system without a long-running transaction
Decide between a migration and a job, and make either one survive being killed halfway

Pick the tool from the row count and from where the new value comes from

Tens of rows

save() in a loop

signals fire, save() runs — all the normal behaviour

Keep it

optimising here trades correctness for nothing

Thousands of rows

bulk_create / bulk_update

with batch_size, and update_conflicts for re-runs

save() and signals are skipped

reproduce that work in bulk too

.iterator() on the read side

the write bound does not bound the read

Value needs no Python

qs.update(f=F("f") + 1)

one statement, no instances, no race

Deletes are the exception

batch them, one transaction per batch

Millions of rows

Resume from the last pk

durable marker, re-enqueue, survive a deploy

Migration does schema only

elidable RunPython for anything small enough to stay

Bulk writes, and what they skip

Collapsing thousands of statements into a handful, and the behaviour that disappears when you do.

`bulk_create()`, `bulk_update()`, and what they skip

coreadvanced

Calling `save()` in a loop over 50,000 objects issues 50,000 statements. `bulk_create()` and `bulk_update()` collapse that into a handful, and `batch_size` controls how many rows go in each one. The price is that they are deliberately dumb: Django's documentation states that "the model's `save()` method will not be called, and the `pre_save` and `post_save` signals will not be sent". Anything your `save()` override does — stamping a field, updating a search vector, writing an audit row — silently does not happen. That is a feature when you want raw speed and a serious bug when you did not know about it.

Think of it as

Think of these as writing rows rather than saving objects. Every convenience the ORM normally layers on top of a write lives in `Model.save()` and in the signals it fires, and both are bypassed here by design — that is where the speed comes from. So the question before using them is always "what else was happening when this object saved?", and the honest way to answer it is to read the model's `save()` and grep for receivers on its signals, rather than to assume. Where the behaviour is genuinely needed, the fix is not to abandon bulk writes but to do the same work in bulk too: compute the derived fields in the loop that builds the objects, and write the audit rows with their own `bulk_create`. `batch_size` is the second half. It bounds how many rows go into one statement, and it matters for two different reasons: a very large statement can exceed what the database or driver will accept, and for `bulk_update` the generated SQL grows with the number of objects because each one contributes its own `WHEN` clause. Django's docs also warn that `bulk_update()` "prepares all of the `WHEN` clauses for every object across all batches before executing any queries" — so the SQL text for the entire operation is built up front, and a very large list is a memory cost even with a small batch size. The remaining rule is about identity: on PostgreSQL `bulk_create` can return primary keys, but not when `ignore_conflicts` is on, and `bulk_update` cannot change a primary key at all.

python

Order.objects.bulk_create(objs, batch_size=500)
Order.objects.bulk_update(objs, ["total", "state"], batch_size=500)

What we're doing: Import 200,000 rows safely: chunked, re-runnable, and with the derived values the skipped `save()` would have set.

imports/orders.pypython

def import_orders(path, batch=1000):
    created = 0

    for chunk in batched(read_rows(path), batch):        # never the whole file
        orders, audits = [], []

        for row in chunk:
            order = Order(
                reference=row["reference"],
                total=Decimal(row["total"]),
                slug=slugify(row["reference"]),           # save() would have done this
                search_text=f"{row['reference']} {row['customer']}".lower(),
            )
            orders.append(order)

        written = Order.objects.bulk_create(
            orders,
            batch_size=batch,
            update_conflicts=True,                        # re-running the import is safe
            update_fields=["total", "search_text"],       # created_at is deliberately absent
            unique_fields=["reference"],
        )
        created += len(written)

        audits = [
            AuditEntry(model="Order", key=o.reference, action="import")
            for o in orders
        ]
        AuditEntry.objects.bulk_create(audits, batch_size=batch)   # the signal's work, in bulk

    return created


def recalculate_totals():
    """bulk_update: change loaded objects without 50,000 UPDATE statements."""
    changed = []

    for order in Order.objects.filter(needs_recalc=True).iterator(chunk_size=2000):
        order.total = recalculate(order)
        changed.append(order)

        if len(changed) >= 1000:
            Order.objects.bulk_update(changed, ["total"], batch_size=1000)
            changed.clear()

    if changed:
        Order.objects.bulk_update(changed, ["total"], batch_size=1000)

11–12: The two fields the model's `save()` normally derives, computed here instead. This is the honest way to use a bulk write: replace the skipped behaviour rather than lose it.
18–21: The upsert. `unique_fields` must match a real unique constraint, and leaving `created_at` out of `update_fields` is what stops a re-run rewriting the original import time.
25–29: The audit rows the `post_save` receiver would have written, done as one more bulk insert. One extra statement per chunk instead of one per row.
37–38: Reading with `.iterator()` and writing with `bulk_update()` — the read side and the write side each need their own bound, and neither one fixes the other.
41–43: Flushing at 1,000 keeps both the accumulator and the generated SQL bounded. Django builds all the `WHEN` clauses up front, so a single call with 200,000 objects is a large memory cost regardless of `batch_size`.

Why this works: The import is bounded on input, safe to re-run, and reproduces both pieces of behaviour the bulk path skips — which is what makes it a replacement for the loop rather than a faster way to lose data quality.

Assuming `bulk_create` runs your `save()` override

Wrong

python

class Order(models.Model):
    def save(self, *args, **kwargs):
        self.slug = slugify(self.reference)      # never runs below
        super().save(*args, **kwargs)

Order.objects.bulk_create(orders)                # 200,000 rows with slug = ""

Better

python

orders = [
    Order(reference=r["reference"], slug=slugify(r["reference"]))
    for r in rows
]
Order.objects.bulk_create(orders, batch_size=500)

What you see: The import reports success and a fifth of the site 404s, because every imported record has an empty slug. The rows are present and look fine in the admin, so the cause is not obvious for some time.

Why: Django states the caveat plainly: `save()` is not called and `pre_save`/`post_save` are not sent. Everything an application layers onto saving — derived fields, denormalised counters, search vectors, audit trails, cache invalidation — lives in exactly those two places, so a bulk write silently skips all of it. The rule is to read the model's `save()` and check for signal receivers before switching, then reproduce whatever they did in the loop that builds the objects.

Every argument is a decision — and one of them is silent

Order.objects.bulk_create(new_orders, batch_size=500, update_conflicts=True, update_fields=["total"], unique_fields=["reference"])

bulk_create

skips save() and signals — The documented caveat: "the model's save() method will not be called, and the pre_save and post_save signals will not be sent." Read the model's save() before choosing this.

new_orders

unsaved instances, built in memory — The list itself is a memory cost. For very large inputs, build and write it in chunks rather than assembling millions of objects first.

batch_size=500

rows per statement — Bounds statement size so the database or driver does not reject it. Without it Django uses one statement for everything, which can be very large.

update_conflicts=True

upsert instead of error — A conflicting row is updated rather than raising `IntegrityError`. This is what makes a re-run of an import safe.

update_fields=["total"]

what a conflict overwrites — Only these columns are updated on conflict. Omitting a field means an existing row keeps its current value for it — which is usually what you want for `created_at`.

unique_fields=["reference"]

what counts as a conflict — Must match a real unique constraint on the table. This is the column the database uses to decide "the same row", so it defines what a re-run means.

Which write to reach for

Which write to reach for
Goal	Use	Loses
insert many new rows	`bulk_create(objs, batch_size=…)`	`save()`, signals, m2m, MTI children
insert, ignoring duplicates	`bulk_create(..., ignore_conflicts=True)`	the above and returned primary keys
insert or update on a key	`bulk_create(..., update_conflicts=True, update_fields=…, unique_fields=…)`	`save()`, signals
change loaded objects	`bulk_update(objs, ["field"], batch_size=…)`	`save()`, signals; cannot touch the pk
set a value for every matching row	`qs.update(field=…)`	`save()`, signals — and never loads the rows

Together

python

Order.objects.bulk_create(
    orders, batch_size=500,
    update_conflicts=True, update_fields=["total"], unique_fields=["reference"],
)

Remember: Bulk writes write rows; they do not save objects. `save()`, `pre_save` and `post_save` are all skipped, which is where the speed comes from and where the data-quality bugs come from — so read the model's `save()` and its signal receivers first, then reproduce that work in the loop that builds the objects, in bulk. Use `update_conflicts` with `unique_fields` to make an import re-runnable, and keep `created_at` out of `update_fields`. Bound both sides: `.iterator()` on the read, a flush counter on the write, because `bulk_update` builds every `WHEN` clause before it executes anything.

Changing rows without loading them

Database-side updates with `F()`, and why deletion is the one operation that has to be batched.

Database-side updates, and deleting in batches

coreadvanced

The cheapest way to change a million rows is not to load them. `qs.update(...)` is "performed at the SQL level" — one statement, no rows fetched, no Python objects built — and `F()` lets the new value be computed from the old one inside the database. The same idea does not extend to deletes, though: one `DELETE` over a million rows takes locks for the whole operation and produces a very large transaction, so deletes are done in bounded batches instead. Both skip `save()` and signals, exactly like the bulk methods.

Think of it as

Sort a large write by whether the new value depends on Python. If it does not — set a flag, add a fixed amount, copy one column into another, derive a value with a database function — the work belongs in the database and should never travel over the network at all. `qs.update()` and `F()` express that, and their cost is independent of the row count in a way a loop can never be. If the new value genuinely requires Python (a call to a library, an external lookup, arbitrary business logic), the rows have to come back, and then the tools from the previous concept apply: chunked reads and bulk writes. The interesting asymmetry is deletion. An `UPDATE` is a single statement whose duration you can usually accept; a `DELETE` over a huge range holds locks on every affected row until it commits, has to cascade to related tables, and produces a transaction big enough to affect replication and vacuuming. So the pattern is a loop of small deletes, each its own transaction, each bounded by a subquery of primary keys. That also makes the job interruptible: stopping between batches leaves a consistent database and a job you can resume, rather than a rollback of an hour's work. The last thing to hold onto is that `qs.delete()` is not always one statement — Django collects related objects to honour `on_delete`, and a `CASCADE` on a large child table can turn one deletion into a lot of work you did not write.

python

qs.update(field=F("field") + 1)          # computed in the database
qs.filter(pk__in=qs.values("pk")[:5000]).delete()   # one bounded batch

What we're doing: Archive and then purge four years of orders without a long-running transaction, and without loading a row.

orders/management/commands/purge_old_orders.pypython

def archive(cutoff):
    """Database-side update: no rows travel anywhere."""
    return (
        Order.objects
        .filter(placed_at__lt=cutoff, state="paid")
        .update(state="archived", archived_at=Now())     # 1 statement
    )


def purge(cutoff, batch=5000, pause=0.2):
    """Batched delete: many small transactions, interruptible between them."""
    deleted_total = 0

    while True:
        ids = list(
            Order.objects
            .filter(placed_at__lt=cutoff, state="archived")
            .values_list("pk", flat=True)[:batch]        # bound the batch
        )
        if not ids:
            break

        with transaction.atomic():                        # ONE batch per transaction
            OrderLine.objects.filter(order_id__in=ids).delete()   # explicit, not cascade
            deleted, _ = Order.objects.filter(pk__in=ids).delete()

        deleted_total += deleted
        log.info("purge.batch", extra={"deleted": deleted, "total": deleted_total})
        time.sleep(pause)                                 # let replicas and vacuum keep up

    return deleted_total

7: `Now()` is a database function, so the timestamp is set by the database in the same statement — no Python value, and no second pass over the rows.
15–19: The slice produces the batch of primary keys; the delete then filters on `pk__in`. `update()` and `delete()` cannot be called on a sliced queryset directly, which is why the ids are materialised first.
22: The transaction wraps one batch, not the loop. That is the whole point: stopping between batches leaves a consistent database and a resumable job.
23: Deleting children explicitly rather than relying on `CASCADE` keeps the work visible and bounded. A cascade over a large child table is real work the code does not show.
28: A short pause between batches gives replication and autovacuum room. Without it, a purge that "worked" can still cause replica lag that breaks reads elsewhere.

Why this works: The update never loads a row, and the purge never holds a lock longer than one batch — so the operation can run against a live system, be interrupted by a deploy, and resume without losing progress.

Wrapping the whole batched delete in one transaction

Wrong

python

with transaction.atomic():                    # one transaction around everything
    while ids := get_batch():
        Order.objects.filter(pk__in=ids).delete()
# 4,000,000 rows locked until it finishes; a failure at 90% discards all of it

Better

python

while ids := get_batch():
    with transaction.atomic():                # one transaction per batch
        Order.objects.filter(pk__in=ids).delete()

What you see: Locks pile up, replica lag grows for the duration, and a timeout near the end rolls back hours of work — leaving the table exactly as it started so the next attempt has just as much to do.

Why: Batching the statements without batching the transactions keeps every drawback of a single huge delete: the locks are held until the outer transaction commits, the transaction stays enormous, and nothing is durable until the very end. The benefit of batching comes from committing between batches — locks are released, replication catches up, and completed work stays completed. It is also what makes the job safe to interrupt, which matters because a purge long enough to need batching is long enough to meet a deploy.

One million rows, two ways to change them

Load, change, save

+Every row travels to the application and back
+A model instance built for each one
+One `UPDATE` per row
+Read-modify-write: two concurrent runs lose one increment
+Cost grows with the row count in three dimensions at once

Change in place

−No rows fetched, no instances built
−One statement, whatever the row count
−The database computes the new value from the old
−Concurrency-safe: the increment happens where the row is locked
−Skips `save()` and signals — deliberately

Where the work should happen

Where the work should happen
Change	Do it	Cost
set a constant	`qs.update(state="archived")`	one statement, no rows loaded
derive from the old value	`qs.update(views=F("views") + 1)`	one statement, concurrency-safe
derive from another column	`qs.update(total=F("net") + F("tax"))`	one statement
needs Python or an API call	`.iterator()` + `bulk_update()`	chunked read, batched write
delete a large range	loop of `pk__in` batches	many small transactions, interruptible

Together

python

Order.objects.filter(placed_at__year=2019).update(state="archived")
# 1 statement; returns the number of rows matched

Remember: If the new value does not need Python, the rows should never leave the database — `update()` with `F()` is one statement whose cost does not grow the way a loop does, and it closes the read-modify-write race as a side effect. Deletes are the exception: batch them, and put the transaction *inside* the loop so each batch commits, releases its locks, and leaves the job resumable. Delete children explicitly rather than trusting a cascade to stay small, and pause between batches so replicas and vacuum keep up. Both paths skip `save()` and signals, exactly like the bulk methods.

Migrations, jobs, and resumability

Where long work belongs, and the two properties that turn an interruption into a pause.

Data migrations, background workers, and resumable work

coreadvanced

A `RunPython` migration runs inside the deploy, and the deploy has a timeout. Backfilling ten million rows there blocks every deploy behind it and, if it fails partway, leaves you with a migration recorded as unapplied over a table that is already half-changed. The rule that follows: a migration should make the *schema* ready and hand large data work to a background job that can be run, watched, interrupted and resumed. When the work must live in the migration, make it chunked and idempotent, and mark it `elidable=True` so it can be squashed away later.

Think of it as

Ask how long the work takes and what happens if it stops halfway. Migrations are optimised for neither answer: they run in a deploy pipeline, usually inside a transaction on PostgreSQL, and their unit of recovery is the whole migration. A background job is optimised for both — it can log progress, be retried, be run on a schedule, and be stopped. So the split is to put anything whose duration is proportional to data volume into a job, and keep migrations to the schema changes whose duration is proportional to the *schema*. Where a backfill genuinely has to be in a migration — most often because later code depends on the column being populated — write it to be safe when run twice. That means filtering on what is not yet done rather than on everything, so a restarted migration processes only the remainder, and updating in bounded batches rather than one statement. Idempotence is the property that turns "it failed at 70%" from an incident into a re-run. The same reasoning applies to any long job: define a unit of progress, commit it, and record it somewhere durable. The one that matters for memory is that the marker should be a *value* from the data — the last primary key processed — rather than an offset, for the same reason cursors beat `OFFSET` in pagination: an offset shifts when rows are inserted or deleted, so a resumed job can skip or repeat work.

python

class Migration(migrations.Migration):
    atomic = False        # so a chunked backfill can commit in pieces
    operations = [migrations.RunPython(backfill, migrations.RunPython.noop, elidable=True)]

What we're doing: A backfill that survives being killed at 70%, written once as a migration and once as the job it should be at scale.

orders/migrations/0042_backfill_reference.pypython

def backfill(apps, schema_editor):
    Order = apps.get_model("orders", "Order")     # historical model, not the import

    while True:
        batch = list(
            Order.objects
            .filter(reference__isnull=True)        # only what is NOT done yet
            .values_list("pk", flat=True)[:1000]
        )
        if not batch:
            break

        for pk in batch:
            Order.objects.filter(pk=pk).update(reference=Concat(Value("A-"), F("pk")))


class Migration(migrations.Migration):
    dependencies = [("orders", "0041_add_reference_column")]
    atomic = False                                 # commit per batch, not once at the end

    operations = [
        migrations.RunPython(backfill, migrations.RunPython.noop, elidable=True),
    ]


# ---- the same work at 10,000,000 rows: a job, not a migration -----------
@shared_task(bind=True)
def backfill_references(self, last_pk=0, batch=5000):
    rows = list(
        Order.objects
        .filter(pk__gt=last_pk, reference__isnull=True)
        .order_by("pk")
        .values_list("pk", flat=True)[:batch]
    )
    if not rows:
        return {"state": "done", "last_pk": last_pk}

    with transaction.atomic():
        Order.objects.filter(pk__in=rows).update(reference=Concat(Value("A-"), F("pk")))

    BackfillProgress.objects.update_or_create(
        name="order_reference", defaults={"last_pk": rows[-1]},
    )
    backfill_references.delay(last_pk=rows[-1], batch=batch)     # continue where it stopped

2: `apps.get_model()` returns the model as it existed at this point in the migration history. Importing `Order` directly gives you today's fields, and the migration breaks the moment a later one changes the model.
8: Filtering on `reference__isnull=True` is what makes the migration idempotent: a re-run picks up only the rows still outstanding, so being killed at 70% costs 70% of the work, not all of it.
14: The value is computed by the database. No rows are loaded, and the historical model does not need a `save()` that may not exist in this state.
19–22: `atomic = False` and `elidable=True` together: the first lets each batch commit as it goes, the second lets a future squash drop this operation once every environment has run it.
33: The job version resumes from the last primary key, not an offset — the same reason cursor pagination beats `OFFSET`. Inserts and deletes elsewhere cannot make it skip or repeat rows.
40–44: Progress is recorded durably and the task re-enqueues itself. A deploy in the middle costs one batch; the chain picks up from the marker.

Why this works: Both versions share the two properties that make long work survivable — a filter that describes what remains rather than what exists, and a commit per batch — so an interruption is a pause rather than a rollback.

Importing the model directly in a `RunPython`

Wrong

python

from orders.models import Order          # today's model, not the historical one

def backfill(apps, schema_editor):
    for order in Order.objects.all():
        order.reference = f"A-{order.pk}"
        order.save()

Better

python

def backfill(apps, schema_editor):
    Order = apps.get_model("orders", "Order")    # the model as of THIS migration
    Order.objects.filter(reference__isnull=True).update(
        reference=Concat(Value("A-"), F("pk")),
    )

What you see: The migration works today and fails on a fresh database or in CI, with an error about a column that does not exist — because a later migration added a field the imported class expects and this migration runs before it.

Why: Migrations replay history. Running them from scratch means this one executes at a point where the table has the columns of migration 42, while the imported class describes the columns of the latest migration. `apps.get_model()` builds the model from the migration state, so it matches the table that actually exists at that moment. The imported class also carries `save()` overrides and signal receivers written against today's code, which is a second reason it does not belong in a historical migration.

A long backfill, and the two paths out of a failure

queued

start

running — batch n

batch committed, marker saved

interrupted (deploy, OOM, timeout)

resumed from the last marker

complete

end

all progress rolled back

end

Where the work belongs

Where the work belongs
Work	Migration?	Why
add a nullable column	yes	fast, schema-proportional
backfill 10,000 rows	yes, chunked + `elidable`	short enough to sit inside a deploy
backfill 10,000,000 rows	no — a job	duration is data-proportional; the deploy would time out
recompute a derived field nightly	no — a scheduled task	recurring by nature
a one-off export of everything	no — a job	needs progress, retries, and an artifact

Together

python

operations = [
    migrations.RunPython(backfill, migrations.RunPython.noop, elidable=True),
]

Remember: Keep migrations schema-proportional and hand data-proportional work to a job — a migration runs in the deploy path, has a deploy timeout, and its unit of recovery is all-or-nothing. When a backfill must be a migration, use `apps.get_model()` for the historical model, filter on what is *not yet done* so a re-run finishes the remainder, set `atomic = False` so batches commit, and mark it `elidable=True`. For any long job, resume from the last primary key rather than an offset, and record that marker durably.