Filter concepts by levelShowing all levels.

Django · Section 89

Event-Driven Architecture

Level
advanced
Read
36 min
Concepts
3

The section draws one shape: Django emits a domain event, a broker carries it, and billing, notification, analytics and search each react independently. What makes that work is what the message *says*. A command names an action, so the producer must know every consumer and gains a new dependency with each one; an event names a fact that is already true — `order.placed`, past tense — so consumers can be added or removed without the producer changing, and each owns its own rules about what the fact means. The distinction to get right underneath is queue versus topic. A queue delivers each message to exactly one consumer, which distributes work; a topic delivers to every subscriber, which fans out. RabbitMQ puts it plainly for the fanout case — it "just broadcasts all the messages it receives to all the queues it knows" — and the mature arrangement is both together: one topic feeding several subscriptions, each of which is a queue shared by one consumer's worker pool. Choosing a queue where you meant a topic is the dangerous error, because every message genuinely is processed and nothing looks broken while three of your four systems never run. Note also what Django signals are not: they are synchronous and in-process, so a receiver runs inside the request and usually inside the transaction — good for decoupling within a project, and not a bus. Next, the delivery reality. Brokers offer at-least-once, because the acknowledgement is itself a message that can be lost: unacknowledged deliveries are "automatically requeued when the channel … is closed", and the documentation says outright that consumers "must be prepared to handle redeliveries and … be implemented with idempotence in mind". So "exactly-once" is a processing property you provide, by claiming each event in a table with a unique constraint on its id — before doing the work, since the reverse leaves a window where redelivery repeats it. Ordering holds only within a partition and only if you set the key, so tolerate out-of-order arrival with a monotonic rank check rather than trusting it. Retry the transient failures with backoff and jitter; dead-letter the deterministic ones at once, because a retrying poison message blocks everything behind it — and give the dead-letter queue an alert and a replay command, or it is just a slower way to lose data. Finally the dual write: you cannot update your database and publish to a broker atomically, and reordering only chooses which failure you get. The outbox converts the second write into a row in your own database, committed with the change it describes, and a relay publishes it afterwards — republishing on a crash, which is safe because consumers already deduplicate. The lag that follows is eventual consistency, and where a user can see it, model it. And carry a `version` from the first event you ever publish: add fields rather than changing them, and never redefine a field's meaning, which is the one breaking change no validator can catch.

What is true here

  1. Publish facts in the past tense; commands put every consumer's decision back in the producer.
  2. Topic fans out, queue distributes — and the fan-out mistake is silent.
  3. At-least-once is the contract; deduplicating on the event id is your half of it.
  4. Ordering is per-partition, so set the aggregate key and still tolerate stale events.
  5. The outbox is the only way to make the state change and the event share a fate.

What you will be able to do

  • Design an event a fifth consumer can subscribe to without touching the producer
  • Write a consumer that is correct when the same message arrives three times, out of order
  • Stop a poison message from blocking a partition
  • Publish events that cannot disagree with the state that produced them
From a committed transaction to four consumers — and every place it can go wrong
after COMMITthe mistakefan-outalreadyclaimedolder thancurrentnew andcurrenttransientattempt n+1exhausteddeterministic— do not retry

One transaction

the state change AND the outbox row — they commit together

Publishing inside the transaction

a rollback leaves four consumers acting on nothing

Relay

select_for_update(skip_locked=True) — several processes share it

Topic: order.placed

key = aggregate_id, so one order stays in one partition

Four subscriptions

each a queue, shared by that consumer’s worker pool

Claim: INSERT ProcessedEvent(event_id)

unique constraint — before the work, not after

Duplicate delivery

IntegrityError → acknowledge and stop

Out of order

rank check → ignore, do not regress

Do the work

Transient failure

backoff + jitter, capped

Dead-letter queue

alert + replay command, or it is silent data loss

Acknowledged

  • One transaction — the state change AND the outbox row — they commit together
    • leads to Relay (after COMMIT)
    • on error, leads to Publishing inside the transaction (the mistake)
  • Publishing inside the transaction — a rollback leaves four consumers acting on nothing
  • Relay — select_for_update(skip_locked=True) — several processes share it
    • leads to Topic: order.placed
  • Topic: order.placed — key = aggregate_id, so one order stays in one partition
    • leads to Four subscriptions (fan-out)
  • Four subscriptions — each a queue, shared by that consumer’s worker pool
    • leads to Claim: INSERT ProcessedEvent(event_id)
  • Claim: INSERT ProcessedEvent(event_id) — unique constraint — before the work, not after
    • leads to Duplicate delivery (already claimed)
    • leads to Out of order (older than current)
    • leads to Do the work (new and current)
  • Duplicate delivery — IntegrityError → acknowledge and stop
    • leads to Acknowledged
  • Out of order — rank check → ignore, do not regress
    • leads to Acknowledged
  • Do the work
    • on error, leads to Transient failure (transient)
    • on error, leads to Dead-letter queue (deterministic — do not retry)
    • leads to Acknowledged
  • Transient failure — backoff + jitter, capped
    • leads to Do the work (attempt n+1)
    • on error, leads to Dead-letter queue (exhausted)
  • Dead-letter queue — alert + replay command, or it is silent data loss
  • Acknowledged

Facts, queues and topics

The section's own fan-out shape, and the distinction that decides whether it works at all.

Producers, consumers — and why a topic is not a queue

coreadvanced

The section draws one shape: Django emits a domain event, a broker carries it, and several consumers — billing, notification, analytics, search — react independently. The producer names *what happened* (`order.placed`) rather than what should be done, and does not know who is listening. The distinction that matters most is queue versus topic. A **queue** delivers each message to exactly one consumer, which is how you distribute work. A **topic** delivers each message to every subscriber, which is how you fan out to four systems that each need it.

Think of it as

The change event-driven design asks for is in what a message *says*. A task says "send the confirmation email"; an event says "order 88 was placed". The first names an action, so the producer has to know every action that should follow, and adding a fifth consumer means editing the producer. The second names a fact that is already true, so consumers can be added and removed without the producer changing at all — which is the entire benefit, and the reason to write events in the past tense. It also changes who owns the decision: if billing decides to stop charging on placement, that is billing's change, made in billing. Getting queue and topic straight is the other half, and confusing them produces two opposite failures. Use a queue where you meant a topic and only one of your four consumers sees each event — the other three silently never run, and because each individual message *was* processed, nothing looks broken. Use a topic where you meant a queue and every worker in a pool processes the same job, so the email goes out four times. The rule to hold on to is that fan-out is a property of the *destination*: one topic feeds several subscriptions, and each subscription is a queue serving one pool of workers that share the work. That shape gives you both at once, which is why every mature broker arranges it that way. It is also worth being clear about what Django signals are not. They are synchronous and in-process — the handler runs inside the same transaction and the same request, so a slow one slows the request and a failing one can break the save. They are a good decoupling tool inside a Django project and they are not an event bus; treating them as one gives you all of the coupling problems you were trying to remove plus a new failure mode. The last decision is what goes in the payload, and the useful default is the identifier plus the few fields a consumer needs to decide whether it cares. A fat payload duplicating the whole object becomes a second, stale copy of your data; a payload of only an id forces every consumer to call back and turns one event into four synchronous requests to the service that emitted it.

python
publish("order.placed", {"order_id": order.id, "total": str(order.total)})

What we're doing: Emit a well-shaped domain event from one place, with an envelope consumers can route and deduplicate on.

events/publish.py + orders/services.pypython
# events/publish.py
def build_envelope(topic, payload, *, aggregate_id):
    """Every event carries the same envelope. Consumers depend on this
    shape, not on the payload's internals."""
    return {
        # The consumer's deduplication key. Generated here, once, so a
        # republished event keeps the SAME id and stays deduplicable.
        "event_id": str(uuid4()),
        "topic": topic,
        "version": 1,                      # see the versioning concept
        "aggregate_id": str(aggregate_id), # the ordering key, if the broker partitions
        "occurred_at": timezone.now().isoformat(),
        "payload": payload,
    }


# orders/services.py
def place_order(order, *, actor):
    with transaction.atomic():
        order.transition_to(Order.Status.PLACED, actor=actor)

        # A FACT, in the past tense. Not "send_confirmation_email" — the
        # producer does not decide what anyone does about it.
        OutboxMessage.objects.create(
            **build_envelope(
                "order.placed",
                {
                    # The id, plus what a consumer needs to decide whether
                    # it cares. Not the whole object: a fat payload becomes
                    # a second, stale copy of your data.
                    "order_id": order.id,
                    "customer_id": order.customer_id,
                    "total": str(order.total),      # str(), never float()
                    "currency": order.currency,
                    "item_count": order.items.count(),
                },
                aggregate_id=order.id,
            )
        )

    # Analytics wants every order; billing only wants paid ones. Both get
    # the same event and each decides for itself — that decision belongs
    # in the consumer, which is the whole point of publishing a fact.
    transaction.on_commit(lambda: publish_outbox.delay())
7–9
The event id is generated once, when the event is created. Generating it at publish time would give a republished event a new id, defeating every consumer's deduplication.
11
`aggregate_id` is what a partitioning broker keys on, so all events for one order land in one partition and keep their relative order. Without it, ordering is arbitrary across the whole topic.
22–24
Past tense, and no imperative. `order.placed` lets a fifth consumer appear next year with no change here; `send_confirmation_email` would put that decision back in the producer.
29–35
The middle ground on payload size. Enough for a consumer to filter without calling back, not so much that the event becomes a stale duplicate of the order.
34
`str(order.total)` because JSON has no decimal type — serialising a `Decimal` as a float reintroduces the rounding error the model deliberately avoided.

Why this works: Every consumer receives the same envelope with a stable id and an ordering key, can decide for itself whether an event is relevant, and can be added without the producer changing.

Publishing a command instead of an event

Wrong

python
publish("send_order_confirmation", {"order_id": order.id})
publish("charge_customer", {"order_id": order.id})
publish("reindex_order", {"order_id": order.id})
# the producer now knows, and encodes, every consumer that exists

Better

python
publish("order.placed", {"order_id": order.id, ...})
# one fact; each consumer decides what it means for them

What you see: Every new downstream feature requires a change to the order service, and its code gradually accumulates knowledge of billing, search, analytics and marketing.

Why: A command names an action, so the producer must know which actions should follow — which is the coupling an event bus exists to remove, now expressed through a broker instead of a function call. It also puts each consumer's business rules in the wrong place: "we only charge for orders over £5" belongs in billing, but a `charge_customer` event forces the order service to know it. Publishing the fact inverts the dependency: consumers subscribe to what happened and own their own decisions, and the producer never learns their names.

The section's own shape — one fact, four independent reactions

Broker — one topic, four subscriptions

billing.order-placed

3 workers share this queue

notification.order-placed

2 workers share this queue

analytics.order-placed

1 worker

search.order-placed

2 workers share this queue

Consumers — independent, and each may fail alone

Billing

raises the invoice

Notification

sends the confirmation

Analytics

appends to the warehouse

Search

reindexes the order

  • Django emits order.placed
  • Broker — one topic, four subscriptions — each subscription is itself a queue, shared by that consumer's worker pool
    • billing.order-placed — 3 workers share this queue
    • notification.order-placed — 2 workers share this queue
    • analytics.order-placed — 1 worker
    • search.order-placed — 2 workers share this queue
  • Consumers — independent, and each may fail alone — adding a fifth changes nothing in Django
    • Billing — raises the invoice
    • Notification — sends the confirmation
    • Analytics — appends to the warehouse
    • Search — reindexes the order

Queue or topic — and what going wrong looks like

Queue or topic — and what going wrong looks like
PropertyQueueTopic
each message goes toexactly one consumerevery subscriber
adding a consumersplits the existing workadds an independent copy
use it fordistributing work across a poolnotifying several systems
the wrong choice looks likefour workers all send the emailthree of four systems silently never run
and is caught bya duplicate the customer sees**nothing** — every message was processed

Together

text
order.placed  (topic)
  ├── billing.order-placed        (queue → 3 billing workers share it)
  ├── notification.order-placed   (queue → 2 workers share it)
  ├── analytics.order-placed      (queue → 1 worker)
  └── search.order-placed         (queue → 2 workers share it)

Task, signal or event — three things that are easy to conflate

Task, signal or event — three things that are easy to conflate
MechanismRunsProducer knows the consumer?
a Celery taskasync, another processyes — it names the function
a Django signal**sync, in-process**no, but it shares the request and transaction
a domain eventasync, another serviceno — and does not need to
a direct HTTP callsync, blockingyes, and depends on it being up

Together

python
# An event states a fact. Nothing here knows who reacts.
publish("order.placed", {"order_id": order.id, "total": str(order.total),
                         "currency": order.currency, "placed_at": ...})

Remember: Publish facts in the past tense (`order.placed`), not commands — a command puts every consumer's decision back in the producer. A queue delivers each message to one consumer (work distribution); a topic delivers to every subscriber (fan-out), and the mature shape is one topic feeding several subscriptions that are each a queue for one worker pool. Choosing wrongly in the fan-out direction is the dangerous one, because every message *is* processed and nothing looks broken while three of four systems never run. Django signals are synchronous and in-process — decoupling within a project, never an event bus.

See also: ordering retries and idempotent consumers · eventual consistency outbox and versioning · side effects and when to avoid signals

Advertisement

What delivery actually guarantees

At-least-once, per-partition ordering, retries, and the dead-letter queue that bounds them.

Ordering, at-least-once delivery, and consumers that can be run twice

coreadvanced

Brokers give you **at-least-once** delivery: a message is delivered until it is acknowledged, so a consumer that crashes after doing the work but before acknowledging will see it again. Ordering is guaranteed only within a partition, and only if you gave the broker a key to partition by. So the two properties a consumer needs are that processing the same event twice is harmless, and that arriving out of order does not corrupt state. Retries and a dead-letter queue handle the messages that keep failing.

Think of it as

Exactly-once delivery is not something to configure, and chasing it is the wrong instinct. The acknowledgement is a second network message that can itself be lost, so the broker's only honest choices are to deliver until acknowledged (at-least-once, occasional duplicates) or to acknowledge before processing (at-most-once, occasional loss). Every reliable system picks the first and makes duplicates harmless in the consumer, which is what "exactly-once processing" actually means — the effect happens once even though the message may arrive several times. The consumer-side mechanism is a processed-events table with a unique constraint on the event id: insert first, and if the insert fails the event has already been handled. The constraint is what makes it true, not the check, because two deliveries can be processed concurrently by two workers in the same pool. Ordering deserves being precise about, because "the broker preserves order" is only true within a partition. Events for the same aggregate must therefore share a partition key — the order id, the customer id — or `order.shipped` can genuinely be processed before `order.placed`. Even with a key, ordering across *different* aggregates is not guaranteed and does not need to be. And a consumer should still tolerate the out-of-order case rather than assume the key was set correctly, using the same monotonic rule the webhook section uses: rank the states, ignore anything that moves backwards. Retry policy is the classification that recurs through this batch, with one addition specific to consumers: retries must not block the partition. A message that keeps failing at the head of an ordered partition stops every message behind it, so a poison message becomes an outage for that key. The dead-letter queue is what bounds it — after N attempts, move the message aside, record why, and keep going. The DLQ then needs two things people forget: an alert, because a silent DLQ is data loss with extra steps, and a replay path, because the usual outcome is that you fix a bug and want the failed messages processed after all.

python
UniqueConstraint(fields=["consumer", "event_id"], name="uniq_processed_event")

What we're doing: A consumer that is safe to run twice, tolerant of out-of-order arrival, and gives up into a dead-letter queue instead of blocking the partition.

search/consumers.pypython
class ProcessedEvent(models.Model):
    consumer = models.CharField(max_length=64)
    event_id = models.CharField(max_length=64)
    processed_at = models.DateTimeField(default=timezone.now)

    class Meta:
        constraints = [
            # THE deduplication guarantee. A .exists() check has a gap that
            # two workers in the same pool fit through.
            models.UniqueConstraint(
                fields=["consumer", "event_id"], name="uniq_processed_event"
            ),
        ]


CONSUMER = "search.order-indexer"
RANK = {"placed": 1, "processing": 2, "fulfilled": 3, "cancelled": 3}


@shared_task(bind=True, autoretry_for=(SearchUnavailable,),
             retry_backoff=True, retry_jitter=True,
             retry_kwargs={"max_retries": 5})
def index_order(self, event):
    try:
        with transaction.atomic():
            ProcessedEvent.objects.create(
                consumer=CONSUMER, event_id=event["event_id"]
            )
    except IntegrityError:
        return "duplicate"          # already done; acknowledge and stop

    try:
        doc = SearchDocument.objects.select_for_update().get(
            order_id=event["payload"]["order_id"]
        )
    except SearchDocument.DoesNotExist:
        doc = None

    incoming = event["payload"]["status"]
    if doc and RANK[incoming] < RANK[doc.status]:
        # Out of order: a "placed" arriving after "fulfilled". Ordering only
        # holds within a partition, so a consumer must tolerate this itself.
        return "stale"

    try:
        search_client.upsert(build_document(event["payload"]))
    except SchemaMismatch as exc:
        # Deterministic: attempt 5 fails exactly as attempt 1 did, and a
        # retrying message blocks everything behind it in the partition.
        dead_letter(event, reason=str(exc), consumer=CONSUMER)
        return "dead-lettered"

    return "indexed"
10–12
The unique constraint is the mechanism; the `create()` is just how it is invoked. Without the constraint, two concurrent deliveries both find nothing and both proceed.
24–30
Insert first, then work. Reversing the order — work, then record — means a crash in between reprocesses the event, which is exactly the case at-least-once delivery guarantees will happen.
38–42
The monotonic guard. Even with a correct partition key, a consumer that assumes ordering breaks the first time a key is missing or a partition is rebalanced — so tolerate it rather than depend on it.
44–50
A schema mismatch is dead-lettered immediately rather than retried. Retrying a deterministic failure five times wastes attempts and, on an ordered partition, holds up every message behind it.

Why this works: A redelivered event is a no-op, an out-of-order event cannot regress the document, a poison message is set aside instead of blocking the partition, and transient failures still get five spaced attempts.

Recording the event as processed after doing the work

Wrong

python
search_client.upsert(build_document(event))          # work first
ProcessedEvent.objects.create(event_id=event["event_id"])
# a crash between these two lines means the work runs again on redelivery

Better

python
ProcessedEvent.objects.create(event_id=event["event_id"])   # claim first
search_client.upsert(build_document(event))

What you see: Duplicate side effects — two emails, two invoice lines — appearing only after a deploy or a worker restart, which is precisely when redelivery happens.

Why: At-least-once delivery guarantees the message comes back if the consumer dies before acknowledging, so the window between doing the work and recording it is a window in which the work will be repeated. Claiming the event first inverts that: a crash after the claim but before the work leaves the event marked processed and the work undone, which is a *visible* gap a reconciliation job can find — and a much better failure than an invisible duplicate. If the work must not be lost either, the claim and the work have to share a transaction, which is only possible when the work is a database write.

One message, and every exit from a consumer
duplicatedeliveryout of ordernew, andcurrentsuccessprocess diesredelivered — the deduperow makes this safetransientfailureattempt n+1attemptsexhaustedpermanent failure— do not retry

delivered (at-least-once)

start

INSERT ProcessedEvent(event_id)

IntegrityError — already handled ack and stop

end

rank check: moves backwards? ack and stop

end

do the work

acknowledged — done

end

crash after work, before ack

retry with backoff + jitter

dead-letter queue + alert + replay path

end

  • delivered (at-least-once) (start)
    • → INSERT ProcessedEvent(event_id)
  • INSERT ProcessedEvent(event_id)
    • → IntegrityError — already handled ack and stop when duplicate delivery
    • → rank check: moves backwards? ack and stop when out of order
    • → do the work when new, and current
  • IntegrityError — already handled ack and stop (end)
  • rank check: moves backwards? ack and stop (end)
  • do the work
    • → acknowledged — done when success
    • → crash after work, before ack when process dies
    • → retry with backoff + jitter when transient failure
    • → dead-letter queue + alert + replay path when permanent failure — do not retry
  • acknowledged — done (end)
  • crash after work, before ack
    • → delivered (at-least-once) when redelivered — the dedupe row makes this safe
  • retry with backoff + jitter
    • → do the work when attempt n+1
    • → dead-letter queue + alert + replay path when attempts exhausted
  • dead-letter queue + alert + replay path (end)

The three delivery semantics, and why one of them is not on offer

The three delivery semantics, and why one of them is not on offer
SemanticsMechanismYou get
at-most-onceack before processingloss on a crash — rarely acceptable
at-least-onceack after processingduplicates on a crash — the default
exactly-once *delivery*not achievable: the ack can be lost
exactly-once *processing*at-least-once + dedupewhat people actually mean, and it is your job

Together

python
try:
    ProcessedEvent.objects.create(event_id=event["event_id"])
except IntegrityError:
    return                      # already handled: the constraint said so

Retry classification, and where the message ends up

Retry classification, and where the message ends up
FailureRetryThen
downstream timeout / 5xxyes, backoff + jitterDLQ after N attempts
row not visible yetyes, short delayusually resolves on attempt 2
schema field missingnostraight to the DLQ — retrying is deterministic failure
unparseable messagenoDLQ, and alert: something upstream is broken
already processednoacknowledge and stop — not an error
exhausted attemptsDLQ + alert + a replay path

Together

python
@shared_task(bind=True, autoretry_for=(RequestException,),
             retry_backoff=True, retry_jitter=True,
             retry_kwargs={"max_retries": 5})
def consume(self, raw): ...

Remember: At-least-once is the honest default, because the acknowledgement itself can be lost — so "exactly-once" is a *processing* property you provide, by deduplicating on the event id with a `UniqueConstraint`. Claim the event before doing the work: the reverse leaves a window in which redelivery repeats it. Ordering holds only within a partition and only if you set the key, so tolerate out-of-order arrival with a monotonic rank check rather than trusting it. Retry transient failures with backoff and jitter, dead-letter deterministic ones immediately — a retrying poison message blocks everything behind it — and give the DLQ an alert and a replay command.

See also: producers consumers queues and topics · eventual consistency outbox and versioning · retries dead letter and poison messages

Advertisement

The dual write, and the contract over time

Making the event share a transaction with the change, and versioning it without breaking consumers.

Eventual consistency, the outbox, and versioning an event

coreadvanced

Writing to your database and publishing to a broker are two systems, and there is no transaction spanning both — so a crash between them either loses the event or announces a change that rolled back. The **outbox pattern** removes the gap: write the event as a row in the *same transaction* as the change, and let a separate relay publish it afterwards. Consumers then catch up shortly after, which is **eventual consistency** — a window where the systems legitimately disagree. **Versioning** is what lets the event shape change without breaking them.

Think of it as

The dual-write problem is the thing to see clearly first. `order.save()` then `publish(...)` looks atomic and is not: if the publish fails, the order is placed and nobody is told; if the transaction rolls back after a successful publish, four consumers act on an order that does not exist. Swapping the order does not help, it only changes which failure you get. The outbox works because it converts the second write into a *first-system* write — the event becomes a row in your own database, so it commits or rolls back with the change it describes, and the two can never disagree. Publishing then becomes a separate, retryable step reading committed rows, which is a much easier problem: a relay that crashes just resumes, and at worst publishes something twice, which consumers already tolerate because delivery was at-least-once anyway. Eventual consistency is the consequence, and the honest way to handle it is to make the window visible in the product rather than pretend it is not there. A search index that lags by two seconds is fine; a screen that says "your order is confirmed" while billing has not seen it yet is fine if it does not also promise an invoice number. Where the delay is user-visible, model it — a `pending` status, a "processing" state — which is the same discipline the payment section applies to a confirming payment. Versioning is the part that gets deferred and then hurts. The moment an event has more than one consumer, its shape is a contract, and you cannot deploy all consumers at the same instant. So changes have to be backward compatible by default: adding an optional field is safe, and removing a field, renaming one, or changing its meaning is not. Carry an explicit `version` on the envelope from the first event you ever publish, because adding one later means every consumer must handle its absence. When a breaking change is genuinely needed, publish both versions for a transition period rather than mutating the old one, and remove the old only when you can show nobody is consuming it. The subtlest failure here is a *semantic* change with no schema change at all — `total` switching from excluding tax to including it — which passes every validator and silently corrupts every downstream number. That one needs a new field name, not a new version.

python
OutboxMessage.objects.create(topic="order.placed", payload=…)   # inside atomic()

What we're doing: A relay that publishes committed outbox rows, runs safely in several processes at once, and never loses or reorders an aggregate's events.

events/relay.pypython
class OutboxMessage(models.Model):
    event_id = models.UUIDField(default=uuid4, unique=True)
    topic = models.CharField(max_length=128)
    aggregate_id = models.CharField(max_length=64, db_index=True)
    version = models.PositiveSmallIntegerField(default=1)
    payload = models.JSONField()
    created_at = models.DateTimeField(default=timezone.now)
    published_at = models.DateTimeField(null=True, blank=True, db_index=True)


@shared_task
def publish_outbox(batch_size=200):
    while True:
        with transaction.atomic():
            # skip_locked lets several relay processes run at once: each
            # takes a different batch instead of queueing behind the others.
            rows = list(
                OutboxMessage.objects
                .filter(published_at__isnull=True)
                .order_by("created_at")            # per-aggregate order preserved
                .select_for_update(skip_locked=True)[:batch_size]
            )
            if not rows:
                return

            for row in rows:
                broker.publish(
                    topic=row.topic,
                    key=row.aggregate_id,          # the partition key: order matters per aggregate
                    body={
                        "event_id": str(row.event_id),
                        "version": row.version,
                        "occurred_at": row.created_at.isoformat(),
                        "payload": row.payload,
                    },
                )

            # Marked published INSIDE the transaction that holds the locks.
            # A crash before this commits simply redelivers the batch — and
            # consumers deduplicate on event_id, so that is harmless.
            OutboxMessage.objects.filter(
                pk__in=[r.pk for r in rows]
            ).update(published_at=timezone.now())
14–21
`skip_locked=True` is what turns a single-process relay into a scalable one: a second process takes the next unlocked batch rather than blocking on the first.
20
Ordering by `created_at` keeps events for one aggregate in the order they were produced, which — combined with the partition key below — is what preserves the only ordering guarantee that matters.
28
The aggregate id becomes the broker's partition key, so all events for order 88 land in one partition and arrive in order. Without it, ordering is arbitrary even though the relay published them in sequence.
37–42
The at-least-once trade, made deliberately. Marking rows published after the broker call risks republishing on a crash; marking them before risks losing them. Republishing is the safe direction because consumers already deduplicate.

Why this works: No event can exist without its state change or vice versa, several relay processes can share the work, per-aggregate ordering survives, and a crash costs a duplicate rather than a loss.

Publishing inside the transaction

Wrong

python
with transaction.atomic():
    order.transition_to(Order.Status.PLACED)
    broker.publish("order.placed", {...})     # sent immediately
    charge_customer(order)                    # raises → transaction rolls back
# the order was never placed, but four consumers already reacted

Better

python
with transaction.atomic():
    order.transition_to(Order.Status.PLACED)
    OutboxMessage.objects.create(topic="order.placed", payload={...})
transaction.on_commit(lambda: publish_outbox.delay())

What you see: Downstream systems occasionally hold records for orders that do not exist — invoices for cancelled checkouts, search entries with no backing row — with nothing in the logs to explain them.

Why: A broker publish is not part of your database transaction, so it takes effect immediately and cannot be rolled back. If anything later in the block raises, the database change disappears and the event does not, leaving every consumer with a record of something that never happened. This is worse than losing an event: a missing event is a gap reconciliation can find, while a phantom event actively creates wrong data in four systems. The outbox row is inside the transaction precisely so it shares the rollback.

The dual write, and the outbox that removes it

Above: two systems, no shared transaction — whichever order you choose, a crash in the gap produces an inconsistency. Below: the event is a row in your own database, so it commits with the change, and a relay publishes it afterwards.

  • The top half shows a dual write: Django saves the order to the database, then publishes to the broker. A crash in the gap between them means the order exists but no consumer is ever told.
  • A note records that reversing the order does not help — publishing first means consumers act on a change that may then roll back.
  • The bottom half shows the outbox: one transaction writes both the order row and an outbox row, so they commit together. A separate relay process then reads committed outbox rows and publishes them, retrying safely because consumers already deduplicate.

Why the dual write cannot be made safe by reordering

Why the dual write cannot be made safe by reordering
Order of operationsCrash in between gives
`save()` then `publish()`the change happened; **no consumer ever hears** — silent divergence
`publish()` then `save()`consumers act on a change that **never happened**
`publish()` inside `atomic()`the same, plus a published event for a rolled-back transaction
**outbox row inside `atomic()`**nothing lost: the row and the change share one commit

Together

python
with transaction.atomic():
    order.transition_to(Order.Status.PLACED)
    OutboxMessage.objects.create(topic="order.placed", payload=…)   # same commit

Which event changes are safe

Which event changes are safe
ChangeSafe?How to do it
add an optional fieldyesconsumers ignore what they do not read
add a required fieldnonew version, or default it
rename a fieldnoadd the new name, publish both, remove later
remove a fieldnostop reading it, prove nobody does, then remove
tighten a type (`str` → `int`)nonew version
**change a field's meaning****no — and nothing detects it**a new field name, never a redefinition

Together

python
# Not: total now includes tax.  Instead:
{"total_excluding_tax": "49.00", "total_including_tax": "58.80"}

Remember: You cannot write to your database and a broker atomically, and reordering only changes which failure you get — so make the event a row in the same transaction and let a relay publish it afterwards. Republishing is the safe direction, because consumers already deduplicate. Scale the relay with `select_for_update(skip_locked=True)`, and use the aggregate id as the partition key so per-aggregate order survives. Accept the lag as eventual consistency and model it in the product where a user can see it. Carry a `version` from your very first event, add fields rather than changing them, and never redefine a field's meaning — that is the break nothing detects.

See also: ordering retries and idempotent consumers · producers consumers queues and topics · on commit and durable

Advertisement