beginnerintermediateadvanced

Django · Section 62

Background Jobs

Level: advanced
Read: 24 min
Concepts: 2

The test for moving work out of a request is not whether it is slow but whether the caller needs its result to know their action succeeded. A signup is complete when the user row exists; the welcome email is a consequence. Email, report generation, image and video processing, and notification fan-out all fail that test — they are slow, they depend on something outside your database, and holding the connection makes your response time the sum of every third party you talk to, so a failing welcome email can fail a successful signup. The enqueue itself has one rule that is easy to get wrong: call it from `transaction.on_commit()`, never inside the transaction, because the broker is not part of your transaction and a worker can pick the message up in the milliseconds before `COMMIT` and find no row — and `on_commit` additionally cancels the job for free when the transaction rolls back. Pass a primary key rather than a model instance, since arguments are serialized at publish time and an instance becomes a stale snapshot. Return 202 with a job id and a poll URL rather than a spinner. The second family is bulk and recurring work — imports, exports, scraping, third-party sync, scheduled jobs and backfills — which is sized by data rather than by a request and therefore must be written assuming interruption: chunk it, checkpoint it, make each chunk idempotent, use `iterator()` and `bulk_create(batch_size=)` instead of loading result sets into memory, and stream exports rather than building them in RAM. Scheduled jobs need one thing the others do not, because nobody is waiting on them: alerting on a stale success heartbeat, since a job that stopped running raises no exception at all.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

Move work out when the caller does not need its result to answer honestly — not merely because it is slow.
Enqueue from transaction.on_commit(); .delay() inside a transaction races the commit.
Pass an id, never a model instance — arguments are serialized when the message is published.
Bulk work is sized by data, so chunk it, checkpoint it, and make each chunk idempotent.
A scheduled job fails as an absence: alert on a stale heartbeat, not on exceptions.

What you will be able to do

Decide correctly which work belongs in the request and which belongs in a worker
Enqueue safely around a transaction, and design the endpoint that reports progress
Write imports, exports and backfills that survive being interrupted mid-run
Notice a scheduled job that has silently stopped running

What stays in the request, what leaves it, and how it leaves safely

Request arrives

Does the caller need this result to know it succeeded?

the actual test — not "is it slow?"

Stays inline

validation, the write itself, a payment authorisation the answer depends on

COMMIT, then transaction.on_commit()

never .delay() inside the transaction — the broker is not transactional

201, or 202 + job id and poll URL

answered in milliseconds

Per-request side effects

email · reports · images · video · notifications

Bulk and recurring work

imports · exports · scraping · sync · schedules · backfills

Chunk, checkpoint, make idempotent

written assuming it will be interrupted

Heartbeat + staleness alert

the only way to notice a schedule that stopped firing

Work that leaves the request

The test for moving work out, and the one safe way to enqueue it.

What belongs in a background job

coreintermediate

A request should do the smallest amount of work that lets it answer honestly, and hand everything else to a worker. Sending email, rendering a report, resizing an image, transcoding a video and fanning out notifications all share the same three properties: they are slow, they depend on something outside your database, and the user does not need their result to know their action succeeded. Keeping any of them inline means the response time is the sum of every third party you talk to, and a failure in the least important step — the welcome email — fails the whole request. The pattern is: write the row, return the response, and enqueue the side effect from `transaction.on_commit()`.

Think of it as

The test is not "is this slow?" but "does the caller need this to have finished before I can honestly answer?". A signup is complete when the user row exists; the welcome email is a consequence, not part of the fact. Framing it that way settles most cases immediately and also exposes the ones that are genuinely inline — you cannot enqueue a payment authorisation and tell the user their order is confirmed, because the confirmation depends on it. The second half is *when* to enqueue, and it is the part that goes wrong most often. Calling `.delay()` inside a transaction means the message can reach a worker before the transaction commits, so the worker looks up a row the database will not show it yet. `transaction.on_commit()` defers the enqueue until after the commit, which fixes both directions: the worker never sees a missing row, and a rollback silently cancels the job instead of sending an email about an order that no longer exists. The third thing worth deciding early is what the user sees. "Your report is being prepared" with a job id and a poll endpoint is a better product than a thirty-second spinner, and it is also the only shape that survives a worker restart.

python

transaction.on_commit(lambda: send_welcome.delay(user.pk))
# after COMMIT, and cancelled automatically if the transaction rolls back

What we're doing: A signup and a report request that both answer immediately, with their side effects enqueued safely.

accounts/views.py + reports/views.pypython

class SignupView(APIView):
    def post(self, request):
        serializer = SignupSerializer(data=request.data)
        serializer.is_valid(raise_exception=True)

        with transaction.atomic():
            user = serializer.save()
            transaction.on_commit(lambda: send_welcome_email.delay(user.pk))

        return Response(UserSerializer(user).data, status=201)


class ReportView(APIView):
    def post(self, request):
        with transaction.atomic():
            job = ReportJob.objects.create(
                requested_by=request.user, status=ReportJob.Status.QUEUED)
            transaction.on_commit(lambda: build_report.delay(job.pk))

        return Response(
            {"job_id": str(job.pk), "status": "queued",
             "poll_url": reverse("report-status", args=[job.pk])},
            status=202,
        )

6–8: The enqueue is registered inside the block but *runs* after the commit. Calling `.delay()` directly here would let a worker read `user.pk` before the row is visible.
8: `user.pk`, not `user`. The argument is serialized into the message, so passing the instance means the worker acts on a snapshot that may already be out of date.
16–18: A row representing the job, created before the task is enqueued, is what makes the work observable — the client has something to poll and support has something to look at.
20–24: 202 with a job id and a poll URL. This is a better product than a spinner and it is also the only shape that survives the worker being restarted mid-report.

Why this works: Both endpoints answer in milliseconds, both are safe against a rollback, and both leave a durable record of the work — none of which is true if the side effect runs inline.

Calling `.delay()` inside the transaction

Wrong

python

with transaction.atomic():
    order = Order.objects.create(...)
    send_confirmation.delay(order.pk)      # the worker may run BEFORE the COMMIT

Better

python

with transaction.atomic():
    order = Order.objects.create(...)
    transaction.on_commit(lambda: send_confirmation.delay(order.pk))

What you see: `Order.DoesNotExist` in the worker for an order that plainly exists by the time you look. It happens under load and never in development, because a local worker is slow enough to lose the race that a busy one wins.

Why: A row created inside a transaction is invisible to other connections until `COMMIT`, but the broker is not part of that transaction — the message is available the instant `.delay()` returns. A worker that picks it up in the milliseconds before the commit queries a database that has never heard of the order. `on_commit()` defers the enqueue past the commit, and as a bonus discards it entirely if the transaction rolls back.

The same signup, inline and enqueued

Client

Django

Database

Worker

1. POST /signup/
2. BEGIN; INSERT user
3. inline: connect to SMTP and sendthe response now waits on a third party you do not control
4. SMTP times out after 30 s
5. 500 — and the user row was rolled backa failed welcome email destroyed a successful signup
6. POST /signup/ (enqueued version)
7. BEGIN; INSERT user; COMMIT
8. on_commit → send_welcome.delay(user.pk)after COMMIT, so the worker can always find the row
9. 201 Created — in 40 ms
10. SELECT user 57; send; retry on failurea failure retries instead of destroying the signup

The section's five, and why each one leaves the request

The section's five, and why each one leaves the request
Work	Why it cannot stay inline	What the request returns
Email	an SMTP or API call you do not control	201, with the email queued
Reports	seconds to minutes of querying and rendering	202 + a job id and a poll URL
Image processing	CPU-bound; blocks a worker entirely	201 with the original; thumbnails appear later
Video processing	minutes; often an external service	202 + a status the client polls
Notifications	fan-out to many recipients and channels	200 — the action, not the fan-out

Together

python

with transaction.atomic():
    order = Order.objects.create(...)
    transaction.on_commit(lambda: send_confirmation.delay(order.pk))

Remember: The test is whether the caller needs the result to know their action succeeded — not whether it is slow. Email, reports, image and video processing, and notification fan-out all fail that test, so they leave the request. Enqueue from `transaction.on_commit()`, never inside the transaction, or a worker can look for a row that has not committed; `on_commit` also cancels the job automatically on rollback. Pass an id, never an instance. And return 202 with a job id and a poll URL rather than holding the connection.

Bulk, scheduled, and external work

Jobs sized by data rather than by a request — written assuming they will be interrupted.

Imports, exports, scraping, sync, scheduled jobs, and backfills

coreadvanced

The second family of background work is bulk and recurring rather than per-request. Large **imports** and **exports** are unbounded by definition — the file decides how long they take — so they need chunking, `iterator()` rather than loading everything into memory, and a progress record the user can watch. **Scraping** and **third-party synchronisation** add someone else's availability and rate limits to your job, which makes retries and backoff mandatory rather than nice. **Scheduled jobs** run on a clock (Celery Beat, or the platform's own scheduler) and have a failure mode the others do not: nobody is waiting, so a broken one is silent until someone notices the absence. **Backfills** are the one-off cousin — a data migration too large or too slow to run inside `migrate`.

Think of it as

The unifying property here is that the work is sized by data rather than by a request, so every one of these jobs must be written as though it will be interrupted. That single assumption produces the right design in all six cases: process in chunks so a restart resumes rather than restarts, record progress so "resume" has something to read, and make each chunk idempotent so re-running one is free. It is also why "one task per file" is usually wrong and "one task that enqueues a task per chunk" is usually right — the second shape gives you retries at chunk granularity, parallelism for free, and a job that survives a deploy. Scheduled work adds its own failure mode worth naming separately: a nightly job that stops running produces no error, no alert and no complaint, because its output is an absence. The fix is to alert on the *last success timestamp* rather than on exceptions — a heartbeat the job writes and a monitor that notices when it goes stale. And backfills deserve to stay out of migrations for a practical reason: a migration holds a transaction and blocks deploys, so a multi-hour data rewrite inside one turns a routine release into an outage.

python

@shared_task(bind=True, autoretry_for=(RequestException,), retry_backoff=True,
             retry_backoff_max=600, max_retries=5)
def sync_partner(self, cursor=None):
    ...

What we're doing: A nightly partner sync that resumes from a checkpoint, backs off on their outage, and is noticed when it stops running.

integrations/tasks.pypython

@shared_task(
    bind=True,
    autoretry_for=(requests.RequestException,),
    retry_backoff=True, retry_backoff_max=600, max_retries=5,
)
def sync_partner(self, cursor=None):
    state, _ = SyncState.objects.get_or_create(source="partner")
    cursor = cursor or state.cursor

    response = requests.get(PARTNER_URL, params={"after": cursor}, timeout=20)
    response.raise_for_status()
    payload = response.json()

    with transaction.atomic():
        Product.objects.bulk_create(
            [Product(**row) for row in payload["items"]],
            update_conflicts=True, update_fields=["price", "stock"],
            unique_fields=["sku"], batch_size=500,
        )
        state.cursor = payload["next_cursor"]
        state.last_success = timezone.now()
        state.save(update_fields=["cursor", "last_success"])

    if payload["next_cursor"]:
        sync_partner.delay(payload["next_cursor"])   # continue from the checkpoint


@shared_task
def check_sync_freshness():
    stale = SyncState.objects.filter(last_success__lt=timezone.now() - timedelta(hours=26))
    for state in stale:
        alert(f"{state.source} has not synced since {state.last_success}")

3–4: Retries on the partner's failures only, with exponential backoff capped at ten minutes — a broken schema in the payload should fail loudly, not retry five times.
10: An explicit timeout. Without one a stalled partner holds a worker indefinitely, and the retry policy above never gets a chance to run.
15–19: `bulk_create` with `update_conflicts` is an upsert: re-running a page is idempotent, which is what makes the retry safe.
20–22: The cursor and the timestamp are written in the same transaction as the rows, so a crash cannot advance the checkpoint past work that was not saved.
28–32: The heartbeat check. This is the only thing that notices a nightly job that stopped running — an absence produces no exception for anyone to see.

Why this works: Every property this job needs comes from the same assumption: it will be interrupted. The checkpoint makes it resumable, the upsert makes the resume safe, and the freshness alarm makes a silent stop visible.

Loading the whole queryset into memory for an export

Wrong

python

rows = list(Order.objects.select_related("customer").all())   # 4M rows
csv_bytes = render_csv(rows)
return HttpResponse(csv_bytes, content_type="text/csv")

Better

python

def rows():
    yield header
    for order in Order.objects.select_related("customer").iterator(chunk_size=2000):
        yield serialize(order)

return StreamingHttpResponse(rows(), content_type="text/csv")

What you see: The worker is OOM-killed and the request dies as a 502 with no application traceback — because the process was terminated rather than raising. It works fine in staging, where the table has ten thousand rows.

Why: A queryset without `iterator()` fetches every row and caches it on the queryset, so peak memory is the whole result set plus the rendered output, held simultaneously. `iterator()` streams in chunks and does not cache, and `StreamingHttpResponse` yields as it goes, so memory stays flat regardless of row count. For anything genuinely large, writing to object storage and returning a link is better still — it survives the request timing out.

A backfill written to be interrupted

1 · Add the column, do not fill it

The migration is instant and the deploy is not blocked. Filling three million rows inside migrate holds a transaction and stops the release.

2 · Fan out one task per chunk

A dispatcher enqueues chunks of primary keys. Retries become per-chunk, and a deploy interrupts one chunk rather than the whole backfill.

3 · Make each chunk idempotent

The filter is part of the work, so re-running a chunk that already completed is a no-op rather than a double write.

4 · Watch it, then tighten the schema

Progress is a query, not a guess. Only once the count reaches zero does the column become non-nullable.

Six kinds of bulk work, and what each one needs

Six kinds of bulk work, and what each one needs
Work	Bounded by	Needs
Large import	the uploaded file	`bulk_create(batch_size=)`, per-chunk tasks, a progress row
Large export	the query result	`iterator()`, streaming or object storage, never an in-memory file
Web scraping	someone else's site	timeouts, backoff, a politeness delay, a concurrency cap
Third-party sync	their rate limit	cursor/checkpoint state so a restart resumes
Scheduled jobs	the clock	a heartbeat and an alert on staleness — nobody is waiting
Data backfill	the table size	chunked, idempotent, and outside the migration

Together

python

for chunk in chunked(Order.objects.filter(total_cents__isnull=True)
                          .values_list("pk", flat=True).iterator(2000), 500):
    backfill_chunk.delay(list(chunk))

Remember: Bulk work is sized by data, not by a request, so write every one of these jobs assuming it will be interrupted: chunk it, checkpoint it, and make each chunk idempotent so a retry is free. Use `iterator()` and `bulk_create(batch_size=)` rather than loading result sets into memory, and stream or upload exports instead of building them in RAM. Scraping and third-party sync need timeouts and backoff because their duration is someone else's decision. Alert scheduled jobs on a stale heartbeat, not on exceptions — a job that stopped running raises nothing. And keep large backfills out of migrations.