beginnerintermediateadvanced

Django · Section 64

Queue Semantics

Level: advanced
Read: 24 min
Concepts: 2

A queue offers one of two honest guarantees, and the difference is only *when the message is acknowledged*. At-most-once acknowledges before running, so a crash loses the work silently and nothing is ever duplicated. At-least-once acknowledges after completing, so a crash redelivers and a message may be processed more than once. Exactly-once delivery is not on offer, because the acknowledgement itself can be lost — which is precisely what the section warns against designing around. What is achievable is exactly-once *effect*: pick at-least-once, then deduplicate on the producer's own identity with a unique constraint inside the same transaction as the work, so a second delivery is a no-op. Ordering is a separate and weaker promise than most people assume — it holds only with a single consumer and no retries, and both concurrency and a re-queued failure will deliver an older update after a newer one, so consumers should carry a version and use a conditional update rather than encoding an order they were never promised. The other half of the section is what happens to a message that keeps failing. Retries need two bounds: backoff with jitter so a struggling dependency is not hammered, and a maximum so failure eventually reports. Failures sort into transient (retry), permanent (fail immediately, because the identical payload will fail identically), and exhausted — which is where a dead-letter queue belongs, keeping the message out of the main flow but available to inspect and replay. Without a cap, a poison message occupies a worker forever and hides in retry logs. And worker restarts are routine rather than exceptional: a rolling deploy kills workers mid-task several times a week, which is exactly what `acks_late` plus an idempotent body plus a long enough graceful window exists for.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

At-most-once loses on a crash; at-least-once repeats on a crash; exactly-once delivery is not available.
Exactly-once effect is achievable — deduplicate on a stable identity inside the same transaction as the work.
Ordering holds only with one consumer and no retries; carry a version and use a conditional update.
Sort failures into transient, permanent, and exhausted, and give the third a dead-letter path.
A rolling deploy killing workers mid-task is normal operation, not an incident.

What you will be able to do

State which delivery guarantee a system actually provides, and design for its failure mode
Make a consumer safe against both duplication and reordering, with the right tool for each
Bound retries so a poison message becomes a reportable failure rather than a permanent tenant
Build a dead-letter path that can be diagnosed and replayed, not just drained

One message, every outcome — and which design decision covers each

Message published

When is it acknowledged?

the only real difference between the two guarantees

Before execution · at-most-once

a crash loses it, silently

After execution · at-least-once

a crash redelivers it

Delivered twice

expected, not exceptional

Delivered out of order

concurrency and retries both reorder

Unique constraint in the same transaction

turns duplication into exactly-once effect

Conditional update on a version

turns a stale message into a no-op

Keeps failing

Transient → retry with backoff

bounded by max_retries

Permanent → fail now

the same payload fails identically

Dead-letter queue

kept, inspectable, replayable

Effect applied exactly once

Delivery guarantees and ordering

The two real guarantees, why exactly-once delivery is not one of them, and what to do about both duplicates and reordering.

At-most-once, at-least-once, duplicates, and ordering

coreadvanced

A queue offers one of two honest guarantees, and the difference is *when the message is acknowledged*. **At-most-once** acknowledges before running: a crash loses the work, and nothing is ever duplicated. **At-least-once** acknowledges after completing: a crash redelivers, so a message may be processed more than once. There is no third option available in practice — "exactly-once delivery" is not something a network can provide, because the acknowledgement itself can be lost. What you *can* build is exactly-once **effect**, by making the consumer idempotent so a second delivery changes nothing. Ordering is a separate and weaker promise than most people assume: with more than one consumer, or with retries, messages routinely arrive out of order.

Think of it as

Both guarantees are the same bet placed on opposite sides of a coin you cannot see: acknowledge early and you bet the worker survives; acknowledge late and you bet you can tolerate a repeat. Neither bet can be avoided, so the only real decision is which failure your work can absorb — and duplication is almost always the one to choose, because you can engineer around it while lost work is simply gone. That is what the section's closing instruction means: designs that assume exactly-once execution are assuming a guarantee nothing in the stack is providing, and they fail rarely enough that the assumption survives review and long enough for the failure to be expensive. Ordering deserves the same scepticism. A single queue with a single consumer processes in order; add a second worker and messages are handled concurrently, add a retry and a failed message is re-queued behind newer ones, and any of those makes "the update after the create" arrive first. So do not encode ordering assumptions into consumers — carry a version or a timestamp and let each message decide whether it is still relevant, or key work so that a stale message is a no-op.

python

@shared_task(acks_late=True)
def handle(event_id):
    with transaction.atomic():
        ProcessedEvent.objects.create(external_id=event_id)   # unique constraint
        apply(event_id)

What we're doing: A consumer that is safe against both duplication and reordering — the two things at-least-once actually gives you.

sync/tasks.pypython

class ProcessedEvent(models.Model):
    external_id = models.CharField(max_length=128, unique=True)
    received_at = models.DateTimeField(auto_now_add=True)


@shared_task(acks_late=True, max_retries=5)
def apply_product_event(event_id, sku, version, payload):
    # 1. Duplication: the unique constraint IS the check.
    try:
        with transaction.atomic():
            ProcessedEvent.objects.create(external_id=event_id)
            _apply(sku, version, payload)
    except IntegrityError:
        logger.info("event_duplicate", extra={"event_id": event_id})
        return


def _apply(sku, version, payload):
    # 2. Reordering: a conditional update, so a stale message is a no-op.
    updated = (Product.objects
               .filter(sku=sku, version__lt=version)
               .update(version=version, **payload))
    if not updated:
        logger.info("event_stale", extra={"sku": sku, "version": version})

2–3: The producer's own event id, not a timestamp or a hash of the payload — two genuinely distinct events can carry identical contents.
10–12: The claim and the work share a transaction, so a crash between them rolls both back and the redelivered message legitimately runs again.
13–15: The `IntegrityError` is the duplicate path, and returning normally acknowledges it — retrying would repeat work already done.
20–24: `version__lt=version` is the reordering guard: an older event arriving after a newer one matches no rows and quietly does nothing.

Why this works: At-least-once gives you two problems, not one — the same message twice, and messages out of order — and they need different fixes: a unique constraint for the first, a conditional update for the second.

Assuming messages arrive in the order they were sent

Wrong

python

@shared_task(acks_late=True)
def apply_product_event(sku, payload):
    Product.objects.filter(sku=sku).update(**payload)   # last writer wins
# create(price=10) then update(price=20): with two workers, either may land last.

Better

python

@shared_task(acks_late=True)
def apply_product_event(sku, version, payload):
    Product.objects.filter(sku=sku, version__lt=version).update(version=version, **payload)

What you see: Prices and statuses occasionally revert to an earlier value. It correlates with worker count and with retries, so it appears when you scale up and never in a single-worker environment.

Why: Ordering is only guaranteed with one consumer and no retries. Two workers pull from the same queue concurrently, and a message that failed and retried re-enters behind messages published after it — so an older update can be applied last. Carrying a version and refusing to apply anything not strictly newer makes each message decide its own relevance, which is the only approach that survives concurrency.

The same crash, under each guarantee

At-most-once — ack first

+The broker forgets the message before the work starts.
+A worker killed mid-task loses it: no error, no retry, no log line.
+Nothing is ever processed twice.
+Fine for a metrics ping or a cache warm — work that is cheap to lose.
+Wrong for anything with a customer-visible effect.

At-least-once — ack after

−The broker keeps the message until the task returns.
−A worker killed mid-task means redelivery, not loss.
−The task may therefore run twice — that is the price.
−Deduplicate on a stable id to get exactly-once effect.
−The right default for anything that matters.

The two real guarantees, and the one that is not on offer

The two real guarantees, and the one that is not on offer
Guarantee	Acknowledged	On a crash	Consumer must be
At-most-once	before execution	the work is lost	nothing special
At-least-once	after execution	the work is repeated	idempotent
Exactly-once delivery	—	—	not available over a network
Exactly-once effect	after execution	repeated, then deduplicated	idempotent — this is the achievable goal

Together

python

@shared_task(acks_late=True)          # at-least-once
def apply_event(event_id):
    with transaction.atomic():
        ProcessedEvent.objects.create(external_id=event_id)   # unique -> dedupe
        do_the_work(event_id)

Remember: A queue gives you at-most-once (ack first, lose on crash) or at-least-once (ack after, repeat on crash) — never exactly-once delivery, because an acknowledgement can be lost. Choose at-least-once and engineer exactly-once *effect*: deduplicate on the producer's own identity, with a unique constraint inside the same transaction as the work. Then handle the second problem separately — ordering holds only with one consumer and no retries, so carry a version and let a conditional update make a stale message a no-op.

When a message keeps failing

Bounded retries, poison messages, dead-letter queues, and worker restarts as routine.

Retries, dead-letter queues, poison messages, and worker restarts

coreadvanced

A retry is a second attempt at the same message, and it needs two bounds: a **backoff** so attempts spread out instead of hammering a failing dependency, and a **maximum** so failure eventually means failure. A **poison message** is one that will never succeed — malformed input, a reference to a deleted row, a bug — and without a cap it is retried forever, occupying a worker and burying the queue. A **dead-letter queue** is where a message goes once it has exhausted its retries: out of the main queue so work keeps flowing, and kept rather than discarded so someone can inspect and replay it. **Consumer failures** and **worker restarts** are the ordinary case a deploy produces several times a day, and they are exactly what `acks_late` makes survivable.

Think of it as

Sort failures into three buckets and the design follows. *Transient* — a timeout, a 503, a deadlock — retry with backoff, because the same message can succeed later. *Permanent* — a malformed payload, a 400, a `KeyError` — fail immediately, because retrying is four more identical failures plus a delay before anyone is told. *Unknown* — retried the maximum number of times and still failing — dead-letter it, because at that point you do not know which of the first two it is and the queue must keep moving. That third bucket is the one most projects skip, and skipping it has a specific cost: without a dead-letter path, an unretryable message either loops forever or is silently dropped, and both are worse than a queue you can look at. Worker restarts belong in this concept rather than the previous one because they are not an exception — a rolling deploy terminates every worker, several times a week, usually mid-task. Designing for that means `acks_late` so the message survives, a soft time limit so a long task can finish or fail tidily inside the shutdown window, and a graceful timeout long enough for your slowest task. Everything else follows from accepting that a worker being killed mid-task is normal operation.

python

class DeadLetterTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        DeadLetter.objects.create(task=self.name, task_id=task_id,
                                  args=args, kwargs=kwargs, error=str(exc))

What we're doing: A base task class that dead-letters anything which exhausts its retries, plus a replay path.

common/tasks.py + common/admin.pypython

class DeadLetter(models.Model):
    task = models.CharField(max_length=200)
    task_id = models.CharField(max_length=64, unique=True)
    args = models.JSONField(default=list)
    kwargs = models.JSONField(default=dict)
    error = models.TextField()
    traceback = models.TextField(blank=True)
    attempts = models.PositiveSmallIntegerField(default=0)
    replayed_at = models.DateTimeField(null=True, blank=True)


class DeadLetterTask(Task):
    """Base class: anything that exhausts its retries is kept, not dropped."""

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        DeadLetter.objects.update_or_create(
            task_id=task_id,
            defaults={
                "task": self.name, "args": list(args), "kwargs": dict(kwargs),
                "error": f"{type(exc).__name__}: {exc}",
                "traceback": str(einfo)[:8000],
                "attempts": self.request.retries + 1,
            },
        )
        super().on_failure(exc, task_id, args, kwargs, einfo)


@shared_task(base=DeadLetterTask, acks_late=True,
             autoretry_for=(requests.ConnectionError,),
             retry_backoff=True, retry_backoff_max=600, max_retries=5)
def notify_partner(order_id):
    ...


@admin.action(description="Replay selected dead letters")
def replay(modeladmin, request, queryset):
    for row in queryset.filter(replayed_at__isnull=True):
        current_app.send_task(row.task, args=row.args, kwargs=row.kwargs)
        row.replayed_at = timezone.now()
        row.save(update_fields=["replayed_at"])

1–9: The row keeps everything needed to diagnose *and* to replay: arguments, the exception, the traceback and the attempt count. A DLQ you cannot act on is just a slower drop.
15–24: `on_failure` runs after retries are exhausted, so this captures only genuinely dead work — not every transient blip on the way.
22: `self.request.retries` is zero-based, so the recorded attempt count is one more than it. Getting this wrong makes a DLQ row look like it was never retried.
28–30: A finite `max_retries` is what makes the dead-letter path reachable at all — `max_retries=None` means a poison message is retried forever and never arrives here.
35–40: Replay from the admin, marking each row so a double-click cannot re-run the same work twice.

Why this works: Bounded retries plus a durable dead-letter row turns "the queue is stuck" into a list someone can read, fix and replay — which is the difference between an incident and a ticket.

`max_retries=None` on a task that can receive bad input

Wrong

python

@shared_task(autoretry_for=(Exception,), retry_backoff=True, max_retries=None)
def import_row(row):
    Product.objects.create(sku=row["sku"], price=Decimal(row["price"]))
# One malformed price is now retried forever.

Better

python

@shared_task(base=DeadLetterTask, autoretry_for=(OperationalError,),
             retry_backoff=True, max_retries=5)
def import_row(row):
    Product.objects.create(sku=row["sku"], price=Decimal(row["price"]))

What you see: One worker child is permanently busy and queue depth grows steadily. The logs show the same traceback every few minutes for days, and because it is a warning-level retry rather than an error, no alert ever fired.

Why: A poison message is one that cannot succeed, and unlimited retries turn it into a permanent tenant of a worker. Combined with `autoretry_for=(Exception,)` — which catches the deterministic failure that makes it poison in the first place — the loop can never end. A finite cap converts it into a single reportable failure, and a dead-letter row makes that failure something a person can look at instead of something that hides in retry logs.

Every path a message can take out of a queue

Queued

start

Running in a worker

Succeeded · acknowledged

end

Transient failure (timeout, 503, deadlock)

Permanent failure (bad payload, 400)

Waiting out the backoff

max_retries exhausted

Dead-letter queue — kept, inspectable

end

Worker killed mid-task (deploy, OOM)

Poison: no cap, retried forever

end

Three failure classes, three destinations

Three failure classes, three destinations
Class	Examples	Action	Bound by
Transient	timeout, 503, deadlock, connection reset	retry with backoff	`max_retries`
Permanent	malformed payload, 400, `KeyError`, deleted row	fail immediately	not retried at all
Exhausted	retried `max_retries` times, still failing	dead-letter it	human or replay job
Interrupted	worker killed by a deploy or an OOM	redelivered by `acks_late`	idempotency

Together

python

def on_failure(self, exc, task_id, args, kwargs, einfo):
    DeadLetter.objects.create(task=self.name, task_id=task_id,
                              args=args, kwargs=kwargs, error=str(exc))

Remember: Bound retries twice — backoff with jitter so a struggling dependency is not hammered, and a maximum so failure eventually reports. Sort failures into transient (retry), permanent (fail now), and exhausted (dead-letter). A poison message with no cap occupies a worker forever and hides in retry logs; a dead-letter row that stores the arguments, the exception and the attempt count is what makes it actionable. And treat worker restarts as routine, because a rolling deploy is: `acks_late`, an idempotent body, and a graceful window longer than your slowest task.