beginnerintermediateadvanced

Python · Background jobs

Background Jobs and Task Queues

Moving slow work off the request path — the queue/worker split, retries, scheduling, and the delivery guarantees a worker must design around

API → Queue → Workerenqueue returns immediately; a separate process does the work
Exponential backoffdelay doubles each retry, capped by max_attempts
Idempotency keymakes a redelivered job safe to run again
Worker concurrencythreads/async for I/O-bound jobs, processes for CPU-bound

1.Task queue tools landscape

Celery, RQ, and Dramatiq are Python task queue libraries that hand work to background workers. Redis and RabbitMQ are the message brokers they run on top of; Kafka is a log-based streaming platform sometimes used the same way.

Key features

Celery is the most feature-complete Python task queue: retries, scheduling, chains, and multiple broker backends
RQ (Redis Queue) is deliberately simpler than Celery — Redis-only, minimal configuration, easier to read the source of
Dramatiq positions itself as a simpler, safer-by-default Celery alternative, with automatic retries built in
Redis and RabbitMQ are message brokers — they store and deliver the messages a task queue library sends between processes
Kafka is a distributed log, not a traditional queue — messages persist and can be replayed, not just consumed once

Six technologies from the roadmap, and the layer each occupies

Six technologies from the roadmap, and the layer each occupies
Name	Layer	What it is for
Celery	task queue library	the most widely used Python distributed task queue; supports Redis, RabbitMQ, and other brokers
RQ	task queue library	a simpler, Redis-only task queue — smaller API surface than Celery
Dramatiq	task queue library	a Celery alternative with automatic retries and a simpler configuration model
Redis	message broker / store	an in-memory data store often used as a broker for Celery, RQ, or Dramatiq
RabbitMQ	message broker	a dedicated message broker implementing AMQP; a common Celery broker choice
Kafka	distributed log / streaming platform	a durable, replayable event log — used for streaming pipelines as well as task-like workloads

Command

python

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task
def my_task(x, y):
    return x + y

Celery/RQ/Dramatiq are task queue libraries; Redis/RabbitMQ/Kafka are the brokers underneath them — pick the library first, then a broker it supports.

2.Job queues and workers

A job queue holds work items until something is free to run them. A worker is a separate process that pulls a job off the queue, runs it, and moves to the next one — decoupling "request the work" from "do the work" in time.

Key features

A job (or task) is a unit of work: a function name plus the arguments it needs, serialized onto the queue
The producer (often a web request handler) enqueues a job and returns immediately, without waiting for it to run
A worker is a long-running process that dequeues jobs one at a time and executes them
Multiple workers can consume the same queue in parallel, so throughput scales by adding worker processes
The queue durably holds a job between enqueue and dequeue — a crashed API process does not lose an already-enqueued job

The four stages a job passes through

The four stages a job passes through
Stage	What happens
Enqueue	the producer serializes the job (function + args) and puts it on the queue
Dequeue	an idle worker pulls the next job off the queue
Execute	the worker runs the job's function with its arguments
Acknowledge	the worker tells the queue the job finished, so it is not redelivered

API enqueues; a separate worker dequeues and runs the job

enqueuedequeueexecute

API

enqueues, returns immediately

Queue

durably holds pending jobs

Worker

dequeues, runs one job at a time

Database / external service

the job's actual side effect

Command

python

import queue, threading

work_queue = queue.Queue()

def worker():
    while (job := work_queue.get()) is not None:
        ...  # run job
        work_queue.task_done()

A queue durably holds jobs between "request the work" and "do the work"; a worker is the separate process that actually does it — that gap is what makes the API response fast.

3.Retries and exponential backoff

A retry re-runs a failed job instead of giving up immediately. Exponential backoff doubles the wait between each retry, so a struggling dependency gets progressively more breathing room instead of being hit again instantly.

Key features

A retry only helps for a transient failure — a network blip, a momentary timeout — not a permanent bug in the job itself
Exponential backoff multiplies the delay each attempt, typically by 2×: 1s, 2s, 4s, 8s...
A max_attempts cap is required — without one, a permanently failing job retries forever
Jitter (a small random amount added to each delay) prevents many workers from retrying in lockstep after a shared outage
A retried job must be safe to run more than once — see idempotency — or a retry can cause duplicate side effects

How the delay grows across attempts, base_delay=1s

How the delay grows across attempts, base_delay=1s
Attempt	Delay before this attempt	Formula
1	0s (first try)	—
2	1s	base_delay * 2^0
3	2s	base_delay * 2^1
4	4s	base_delay * 2^2
5	8s	base_delay * 2^3

Delay doubles each attempt, capped by max_attempts

attempt 1
runs immediately
0s wait
attempt 2
waits 1s
base_delay * 2^0
attempt 3
waits 2s
base_delay * 2^1
attempt 4
waits 4s
base_delay * 2^2 — max_attempts reached, raise if still failing

Command

python

import functools, time

def retry_with_backoff(max_attempts=4, base_delay=1):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except RuntimeError:
                    if attempt == max_attempts:
                        raise
                    time.sleep(base_delay * (2 ** (attempt - 1)))
        return wrapper
    return decorator

A retry only helps a transient failure; exponential backoff (delay doubling each attempt, capped by max_attempts) keeps retries from hammering an already-struggling dependency.

4.Dead-letter queues

A dead-letter queue (DLQ) is a separate queue that holds jobs which failed every retry attempt. It exists so a permanently failing job is held for inspection, instead of being lost or retried forever.

Key features

A job moves to the DLQ only after exhausting its configured max retry attempts
The DLQ is a real, separate queue — jobs there stop being auto-retried, but are not deleted
A human or an automated process later inspects a DLQ job, fixes the root cause, and can requeue it
RabbitMQ supports DLQs natively via a queue argument (x-dead-letter-exchange); Celery and cloud queues (e.g. SQS) provide their own DLQ mechanisms
A growing DLQ is itself an alert-worthy signal — it means something is failing systematically, not occasionally

Command

python

def process_with_dlq(jobs, handler, max_attempts=3):
    dead_letter_queue = []
    for job in jobs:
        for attempt in range(1, max_attempts + 1):
            try:
                handler(job)
                break
            except ValueError as exc:
                if attempt == max_attempts:
                    dead_letter_queue.append({"job": job, "error": str(exc)})
    return dead_letter_queue

A dead-letter queue holds a job that exhausted its retries — not deleted, not retried forever, just set aside for a human or automated process to inspect.

5.Job scheduling and delayed jobs

Job scheduling runs a job at a specific time or on a repeating interval, like "every day at midnight." A delayed job runs once, after a relative wait, like "in 10 minutes" — both hold the job back from a worker until its time arrives.

Key features

A delayed job specifies a countdown or an eta (a specific timestamp) — the job runs once, after that point
A scheduled (or "periodic") job runs repeatedly, usually defined with a cron-style expression
Celery's countdown= and eta= arguments delay a single call; celery beat runs periodic tasks on a schedule
RQ's scheduler (rq-scheduler) and Dramatiq's APScheduler integration provide the same two capabilities for those libraries
The broker (or a dedicated scheduler process) — not the worker — is what holds a delayed/scheduled job until its time arrives

Command

python

# Celery: delay a single call (from Celery's own docs)
send_welcome_email.apply_async(args=[user_id], countdown=600)   # in 10 minutes
send_welcome_email.apply_async(args=[user_id], eta=specific_datetime)

# Celery: a periodic (scheduled) task via celery beat
app.conf.beat_schedule = {
    "cleanup-every-midnight": {
        "task": "tasks.cleanup_temp_files",
        "schedule": crontab(hour=0, minute=0),
    },
}

A delayed job runs once after a relative wait (countdown); a scheduled job runs repeatedly on a fixed pattern (cron-style) — both are held back from a worker until their time arrives.

6.Worker concurrency

Worker concurrency is how many jobs a worker runs at the same time. A concurrency of 1 processes jobs strictly one after another; a concurrency of 8 runs up to 8 jobs in parallel using threads, processes, or async tasks.

Key features

Celery's --concurrency flag sets the number of worker child processes (or threads/greenlets, depending on the pool type)
CPU-bound jobs benefit from process-based concurrency — threads share one GIL and cannot use multiple cores for pure Python code
I/O-bound jobs (network calls, disk waits) benefit from thread- or async-based concurrency, since they spend most time waiting, not computing
Higher concurrency raises throughput but also raises memory use and load on shared resources like a database connection pool
Running more worker processes (horizontal scaling) is a separate lever from concurrency within one worker process

Command

python

# Celery: run a worker with 4 concurrent child processes (from Celery's own docs)
# celery -A tasks worker --concurrency=4

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(run_job, jobs))

Concurrency is how many jobs a worker runs at once — match the mechanism to the job: threads/async for I/O-bound work, processes for CPU-bound work, since threads share one GIL.

7.At-least-once delivery, duplicate processing, and idempotency

At-least-once delivery means a broker guarantees a job arrives one or more times, never zero — but "or more" means a worker can see the same job twice (duplicate processing). Idempotency is designing a job so running it twice has the same effect as running it once.

Key features

At-least-once delivery guarantees a job is delivered ≥1 times — never silently dropped, but sometimes duplicated
A duplicate delivery happens when a worker processes a job but crashes, or the network drops, before it can acknowledge — the broker redelivers, assuming it was never handled
Exactly-once delivery is what everyone would prefer, but is extremely hard to guarantee across a network in general — most systems choose at-least-once plus idempotency instead
An idempotency key is a unique identifier for one logical operation (e.g. "charge order #42"), stored so a repeat with the same key is recognized and skipped
A job is idempotent if calling it N times produces the same end state as calling it once — "set balance to 100" is idempotent; "add 10 to balance" is not

Idempotent vs. non-idempotent job actions

Idempotent vs. non-idempotent job actions
Action	Idempotent?	Why
UPDATE orders SET status = 'shipped' WHERE id = 42	yes	setting an absolute value — running it twice leaves the same end state
UPDATE accounts SET balance = balance + 10 WHERE id = 1	no	each run changes the result — running it twice adds 20, not 10
INSERT INTO payments (idempotency_key, ...) ... ON CONFLICT DO NOTHING	yes	the unique key makes a duplicate insert a no-op instead of a second row
send_email(user_id)	no, by default	a duplicate call sends a second email unless deduplicated by a key

Same redelivered job, with and without an idempotency key

Without idempotency

Worker crashes after charging the card, before acknowledging
Broker redelivers the job — it looks unhandled
Handler charges the card again
Customer is billed twice for one order

With an idempotency key

Worker crashes after charging the card, before acknowledging
Broker redelivers the job — it looks unhandled
Handler checks the idempotency key first — already processed
Handler returns the stored result; no second charge

Command

python

processed = {}

def handle_job(idempotency_key, *args):
    if idempotency_key in processed:
        return processed[idempotency_key]   # duplicate delivery — skip
    result = do_the_actual_work(*args)
    processed[idempotency_key] = result
    return result

At-least-once delivery means a worker will sometimes see the same job twice — an idempotency key (tied to the operation, not the delivery attempt) is what makes that safe to run again.

8.Job monitoring and failure recovery

Job monitoring tracks whether jobs are succeeding, failing, or piling up, usually through a dashboard or metrics. Failure recovery is what happens after a job fails for good — alerting someone, and providing a way to requeue it once the root cause is fixed.

Key features

A job passes through states — commonly queued, running, succeeded, failed — and monitoring is tracking which state every job is in
Celery ships Flower, a web dashboard showing active/scheduled/failed tasks and worker status in real time
A queue length that keeps growing (jobs enqueued faster than workers drain them) is an early-warning signal, not just failures
Failure recovery includes: alerting on failure, capturing enough context (job args, error, traceback) to debug without rerunning blind, and a safe way to requeue once the cause is fixed
A job that only logs a failure and does nothing else is monitored but not recoverable — recovery needs an actionable next step, not just a log line

Command

python

job_states = {}

def run_job(job_id, fn):
    job_states[job_id] = {"status": "running"}
    try:
        fn()
        job_states[job_id]["status"] = "succeeded"
    except Exception as exc:
        job_states[job_id]["status"] = "failed"
        job_states[job_id]["error"] = str(exc)

Monitoring tracks what state every job is in (and whether the queue is draining); recovery turns a captured failure into an actionable next step — alert, context, and a safe way to requeue.

Spotted a mistake on this page? Tell us.

Read Sheet