Celery cheat sheet · Django

1.The Celery app, tasks, workers, and task discovery

A Celery deployment has three pieces. The **app** is a `Celery(...)` instance created once in `config/celery.py` and imported from `__init__.py` so it exists before any task is defined. A **task** is a function registered with that app — in a Django project you use `@shared_task`, which registers with whichever app is configured rather than binding to one instance, so reusable apps work. A **worker** is a separate process (`celery -A config worker`) that consumes messages and runs those functions; it is not part of your web server and does not share its memory. **Task discovery** is `app.autodiscover_tasks()`, which imports `tasks.py` from every entry in `INSTALLED_APPS` — which is why a task in a file with any other name is invisible.

Key features

The app object lives in `config/celery.py` and is imported from `config/__init__.py`, so it is created before Django loads any app module
`@shared_task` registers with the current app rather than a specific instance, which is what makes tasks in reusable apps work — prefer it in a Django project
`.delay(*args)` is shorthand for `.apply_async(args=...)`; both publish a message and return an `AsyncResult` immediately, without running anything
`app.autodiscover_tasks()` imports `tasks.py` from every `INSTALLED_APPS` entry — a task defined anywhere else is never registered in the worker
A worker is a separate process with its own imports, settings, database connections and memory; nothing is shared with the web process
`celery -A config worker -l info` prints the registered task list at startup — the fastest way to confirm discovery actually found your task

The three pieces, and where each one lives

The three pieces, and where each one lives
Piece	Where	Created by	Fails as
App	`config/celery.py`	`Celery("config")` + `config.from_object`	tasks register against no app
Task	`<app>/tasks.py`	`@shared_task`	`NotRegistered` in the worker
Worker	its own process	`celery -A config worker`	messages queue up unconsumed
Discovery	`config/celery.py`	`app.autodiscover_tasks()`	silently skips non-`tasks.py` files

What happens between .delay() and the function body running

name notfoundfoundif a backendis configured

View calls send_email.delay(57)

returns in microseconds — nothing has run

Serialize name + args

JSON by default, so arguments must be JSON-safe

Broker (Redis / RabbitMQ)

the message waits here — possibly across a deploy

Worker process

separate imports, separate settings, separate connections

Task registry lookup by name

populated by autodiscover_tasks() from tasks.py only

The function body finally runs

NotRegistered

the task exists in the web process and not in the worker

Result backend (optional)

only if configured — otherwise the return value is discarded

Command

python

@shared_task
def send_welcome_email(user_id):
    ...

send_welcome_email.delay(user.pk)     # publishes a message; does not run anything

The app is created in `config/celery.py` and imported from `config/__init__.py` so it exists before any task is defined. `@shared_task` registers with the current app, which is what makes reusable apps work. `.delay()` publishes a message and returns — the body runs later, in a separate process with its own imports, settings and connections, so pass ids and read current state. And autodiscovery imports `tasks.py` specifically: a task defined anywhere else works in your web process and raises `NotRegistered` in the worker.

2.Brokers and result backends

These are two separate services doing two unrelated jobs, and they are constantly confused because they are usually configured to the same Redis. The **broker** carries messages from your web process to a worker — it is required, and it is where a task lives between `.delay()` and execution. The **result backend** stores what a task *returned*, so somebody can ask for it later; it is optional, and if you never call `.get()` on an `AsyncResult` you do not need one. Storing results you never read is pure cost: every task writes a row or a key that expires on a timer, which on a busy queue is a substantial amount of traffic for data nobody looks at.

Key features

The broker is required and carries the message; the result backend is optional and stores the return value
If nothing calls `AsyncResult.get()`, set `task_ignore_result = True` — otherwise every task writes a result nobody reads
With no result backend configured, results are simply discarded; `.get()` then raises rather than blocking forever
RabbitMQ gives durable queues, acknowledgements and dead-lettering natively; Redis is a data structure server used as a queue and leaves dead-lettering to you
Give the broker its own Redis database — `cache.clear()` issues `FLUSHDB` and will delete queued tasks sharing that database
`result_expires` (one day by default) is what stops a result backend growing without bound

Two services, two failure meanings

Two services, two failure meanings
Property	Broker	Result backend
Required?	yes	no
Holds	the message: task name + arguments	the return value and task state
If it is down	work never happens	work happens; you cannot see the outcome
Typical choice	Redis or RabbitMQ	Redis, or the database, or none
Cost of getting it wrong	lost or undelivered jobs	wasted writes for data nobody reads
Setting	`CELERY_BROKER_URL`	`CELERY_RESULT_BACKEND`

Two services, usually one Redis — and why they need separating

Broker · required

Holds name + arguments

the message waits here, possibly across a deploy

Its own Redis database

cache.clear() runs FLUSHDB and would delete the queue

RabbitMQ adds durability and DLQs

Redis leaves dead-lettering to you

Down means work never happens

delay() raises — decide whether that is a 503

Result backend · optional

Only needed for .get()

most tasks are fired for a side effect, not a value

result_expires bounds growth

one day by default; without it the store grows forever

task_ignore_result = True

the right default; opt in per task

Down means you cannot see outcomes

the work itself completed normally

Command

python

CELERY_BROKER_URL = env("REDIS_URL") + "/0"
CELERY_TASK_IGNORE_RESULT = True

@shared_task(ignore_result=False)      # opt in only where a result is read
def build_report(job_id): ...

The broker is required and carries the message; the result backend is optional and stores the return value. If the broker is down work never happens, while if the backend is down the work still ran and you simply cannot see the outcome. Set `task_ignore_result = True` and opt in per task, because most tasks are fired for a side effect and storing unread results is a write each. Give the broker its own Redis database — `cache.clear()` runs `FLUSHDB` — and on Redis make `visibility_timeout` longer than your slowest task, or it is redelivered mid-run.

3.Retries, backoff, autoretry_for, ETA/countdown, and Beat

A task that talks to anything outside your database will eventually fail for a reason that goes away on its own, which is what retries are for. `self.retry(exc=..., countdown=...)` retries manually and needs `bind=True`; `autoretry_for=(SomeError,)` does it declaratively. `retry_backoff=True` spaces the attempts exponentially instead of hammering a struggling service, `retry_backoff_max` caps the delay (600 seconds by default), and `retry_jitter` — on by default — randomises it so a thousand tasks that failed together do not retry together. `max_retries` defaults to 3. For scheduling *ahead*, `countdown=` and `eta=` delay a single task; **Celery Beat** is the separate process that publishes tasks on a recurring schedule.

Key features

`self.retry()` requires `bind=True`; it raises `Retry`, so code after it does not run, and it counts against `max_retries` (default 3)
`autoretry_for=(Exc,)` plus `retry_kwargs={"max_retries": 5}` is the declarative form — list only exceptions that can succeed on a retry
`retry_backoff=True` gives exponential delays; `retry_backoff_max` caps them at 600 seconds by default; `retry_jitter` is on by default and randomises the delay
Exceeding the limit raises `MaxRetriesExceededError`; `max_retries=None` retries forever, which is almost never what you want
`countdown=n` delays one task by n seconds and `eta=datetime` schedules it for a moment — both hold the message in the broker until then
Celery Beat is a separate process that *publishes* tasks on a schedule; running two Beats fires every periodic task twice

Retry it, or fail it?

Retry it, or fail it?
Failure	Retry?	Why
Connection reset, timeout, 502/503	yes	the input was fine; the world was temporarily not
429 rate limited	yes, with backoff	honour `Retry-After` if the response carries one
400 / 422 from an API	no	the same payload fails identically every time
`ValidationError`, `KeyError`, `TypeError`	no	a bug — retrying hides it behind a delay
`IntegrityError` on a unique key	usually no	often means the work already happened
Database deadlock	yes	the classic transient — retry the whole transaction

One failing task under exponential backoff with jitter

t = 0s
Attempt 1 — partner returns 503
autoretry_for catches RequestException; the task raises Retry rather than failing.
t ≈ 1s
Attempt 2
retry_backoff=True starts at roughly one second. Jitter spreads simultaneous failures apart from here on.
t ≈ 2s
Attempt 3
The delay doubles each time, so a dependency recovering after a few seconds costs almost nothing.
t ≈ 4s
Attempt 4
Still cheap. Most transient failures are already resolved by this point.
t ≈ 8s
Attempt 5
max_retries=5 means this is the last one. Without a cap, "retry forever" hides an outage instead of reporting it.
capped
retry_backoff_max = 600s
For longer-running retry policies the delay stops doubling at ten minutes rather than growing to hours.
exhausted
MaxRetriesExceededError
Now it is a real failure: alert on it, record it, and let a dead-letter path or a human decide what happens next.

Command

python

@shared_task(bind=True, autoretry_for=(RequestException,),
             retry_backoff=True, retry_backoff_max=600,
             retry_jitter=True, max_retries=5)
def notify_partner(self, order_id): ...

Retry only what can succeed unchanged: connection errors, timeouts, 5xx and deadlocks — never a 400, a `ValidationError`, or anything deterministic. Name specific exceptions in `autoretry_for`, because `Exception` turns every bug into a slow one. `retry_backoff` spaces attempts, `retry_backoff_max` caps at 600 seconds, and jitter (on by default) is what stops synchronised retries becoming a thundering herd. `countdown`/`eta` delay one task; Beat publishes on a schedule and must run exactly once.

4.Queues, routing, worker concurrency, and time limits

By default every task goes to one queue named `celery` and every worker consumes it, which means a twenty-minute video transcode sits in front of a password-reset email. **Routing** fixes that: `task_routes` sends tasks to named queues by pattern, and each worker is started with `-Q` naming the queues it consumes, so slow work and fast work stop competing. **Concurrency** is `--concurrency=N`, the number of child processes (or threads, with a different pool) each worker runs — sized by CPU for compute and higher for I/O-bound work. **Time limits** come in two kinds: `task_soft_time_limit` raises `SoftTimeLimitExceeded` inside the task so it can clean up, while `task_time_limit` kills the process outright.

Key features

Everything defaults to one queue named `celery`; without routing, a long task blocks short ones behind it
`task_routes` maps task-name patterns to queues; `apply_async(queue="slow")` overrides per call
A worker consumes only the queues named in `-Q` — start a queue with no worker listening and its tasks simply accumulate
`--concurrency=N` sets child processes for the default prefork pool: roughly one per core for CPU-bound work, higher for I/O-bound
`task_soft_time_limit` raises `SoftTimeLimitExceeded` *inside* the task so it can clean up; `task_time_limit` kills the child process with no chance to run `finally`
`worker_prefetch_multiplier=1` stops a worker reserving several long tasks at once and leaving others idle — the standard setting for slow queues

A three-queue split, and why each differs

A three-queue split, and why each differs
Queue	Typical task	Concurrency	Soft / hard limit	Prefetch
`default`	emails, notifications, webhooks	cores × 2	30 s / 60 s	default
`slow`	reports, exports, transcoding	cores	900 s / 1000 s	1
`maintenance`	nightly reconciliation, backfills	2	3600 s / 3700 s	1

One queue, or three — and where the head-of-line blocking goes

no routingtask_routesconfigureda queue withno -Q worker

Tasks published

emails, reports, transcodes, nightly jobs

One queue: "celery"

the default — everything shares a line

A 20-minute transcode at the head

every password-reset email behind it waits

task_routes by name pattern

the routing decision, made once in settings

default

short tasks · -c 8 · soft limit 30 s

slow

reports and media · -c 4 · prefetch 1 · soft limit 900 s

maintenance

nightly and backfills · -c 2 · off-peak

Worker -Q default

Worker -Q slow

Worker -Q maintenance

A queue nobody consumes

no error anywhere — the tasks just accumulate

Command

python

CELERY_TASK_ROUTES = {
    "media.tasks.*": {"queue": "slow"},
    "billing.tasks.reconcile": {"queue": "maintenance"},
}

Everything lands on one `celery` queue by default, so a long task blocks short ones behind it — split queues by *duration* and give each its own worker, concurrency and limits. A worker only consumes what `-Q` names, and publishing to a queue nobody reads produces no error at all, so monitor queue depth. Size concurrency by CPU for compute and higher for I/O. And always set both limits: the soft one raises inside the task so it can mark itself failed and clean up, while the hard one kills the process and skips every `finally`.

5.Task idempotency, monitoring, recovery, and Flower

Celery acknowledges a message *before* running the task by default, which means a worker killed mid-task loses that work silently. Setting `acks_late=True` moves the acknowledgement to after completion, so a crash causes redelivery instead of loss — and the Celery docs are explicit that this requires the task to be idempotent, because it may now run more than once. Monitoring is the other half: a queue can be deep, a worker can be dead, and tasks can be failing, and none of those raises anything in your application. **Flower** is the standard web UI for Celery's event stream — live workers, task history, queue lengths — and it is a debugging tool rather than an alerting system, so real alerts belong on queue depth, failure rate, and time since last success.

Key features

By default Celery acknowledges a message *before* execution — a worker killed mid-task loses it with no error and no retry
`acks_late=True` acknowledges after the task returns, so a crash redelivers; the Celery docs state this requires the task to be idempotent
Even with `acks_late`, a worker whose *process* is terminated still acknowledges — deliberate, so a crash-looping task does not run forever
Idempotency: record a stable identity with a unique constraint inside the same transaction as the effect, and treat a duplicate as success
Alert on queue depth, failure rate, and last-success age — not on exceptions, because nobody is waiting on a task to notice it failed
Flower reads Celery's event stream for live workers, task history and queue lengths; treat it as a debugger, not as your alerting

Which failure do you want?

Which failure do you want?
Setting	Worker killed mid-task	Requires	Use for
`acks_late=False` (default)	the task is lost, silently	nothing	work that is cheap to lose and expensive to repeat
`acks_late=True`	the task is redelivered	an idempotent body	anything that matters
`acks_late=True` + `reject_on_worker_lost`	requeued even on process loss	a poison-message guard	critical work, with a retry cap

What to watch, and what each signal means

What to watch, and what each signal means
Signal	Means	Alert when
Queue depth	arrival rate exceeds completion rate	depth grows for N minutes
Oldest message age	latency, independent of depth	older than the SLA for that queue
Failure rate	tasks raising after retries	above baseline
Active worker count	a deployment that did not come back	below the expected replica count
Last success age (per schedule)	a periodic task that stopped firing	older than its period + margin

One message under acks_late, including the crash

a workerpicks it upINSERT succeeds— first runIntegrityError— seen beforereturn early:already donework commits,then ackSIGKILL, OOM,node lossnever acknowledged→ redeliveredretriesexhausted

Queued in the broker

start

Reserved by a worker — NOT yet acknowledged

Task body running

Idempotency key inserted (unique constraint)

Key already present — this is a redelivery

Acknowledged · removed from the broker

end

Worker killed mid-task

Raised after max_retries — dead-letter or alert

end

Command

python

@shared_task(acks_late=True, autoretry_for=(RequestException,), max_retries=5)
def charge_order(order_id, key):
    with transaction.atomic():
        TaskRun.objects.create(key=key)     # unique constraint = the duplicate check
        charge(order_id)

Celery acknowledges before running by default, so a killed worker loses the task silently. `acks_late=True` swaps that for redelivery — and the docs are explicit that it requires an idempotent task, so the setting and a unique-constraint claim inside the same transaction as the effect are one decision. Nothing surfaces Celery failures on its own, because nobody is waiting: alert on queue depth, oldest-message age, failure rate and last-success age. Flower is a debugger for when the alert fires, not the alert.