beginnerintermediateadvanced

Django · Event-Driven Architecture

Event-Driven Architecture

Publish facts, tolerate duplicates, and never write to two systems at once

Events are facts, not commandsorder.placed — the producer must not know who reacts
Queue delivers to one; topic delivers to allthe wrong choice silently runs three of four consumers never
At-least-once is the honest defaultexactly-once processing is your job: dedupe on event_id
Ordering holds within a partition onlyset the aggregate id as the key, and still tolerate stale arrivals
The outbox removes the dual writethe event is a row in the same transaction — they commit together

1.Producers, consumers — and why a topic is not a queue

The section draws one shape: Django emits a domain event, a broker carries it, and several consumers — billing, notification, analytics, search — react independently. The producer names *what happened* (`order.placed`) rather than what should be done, and does not know who is listening. The distinction that matters most is queue versus topic. A **queue** delivers each message to exactly one consumer, which is how you distribute work. A **topic** delivers each message to every subscriber, which is how you fan out to four systems that each need it.

Key features

Events name facts in the past tense (`order.placed`), never actions (`send_email`)
Queue: each message goes to exactly one consumer — work distribution
Topic: each message goes to every subscriber — fan-out
The mature shape is both: one topic → several subscriptions, each a queue for a worker pool
Django signals are synchronous and in-process — a decoupling tool, not an event bus
Payload = the id plus the few fields a consumer needs to decide whether it cares

Queue or topic — and what going wrong looks like

Queue or topic — and what going wrong looks like
Property	Queue	Topic
each message goes to	exactly one consumer	every subscriber
adding a consumer	splits the existing work	adds an independent copy
use it for	distributing work across a pool	notifying several systems
the wrong choice looks like	four workers all send the email	three of four systems silently never run
and is caught by	a duplicate the customer sees	nothing — every message was processed

Task, signal or event — three things that are easy to conflate

Task, signal or event — three things that are easy to conflate
Mechanism	Runs	Producer knows the consumer?
a Celery task	async, another process	yes — it names the function
a Django signal	sync, in-process	no, but it shares the request and transaction
a domain event	async, another service	no — and does not need to
a direct HTTP call	sync, blocking	yes, and depends on it being up

The section's own shape — one fact, four independent reactions

Broker — one topic, four subscriptions

billing.order-placed

3 workers share this queue

notification.order-placed

2 workers share this queue

analytics.order-placed

1 worker

search.order-placed

2 workers share this queue

Consumers — independent, and each may fail alone

Billing

raises the invoice

Notification

sends the confirmation

Analytics

appends to the warehouse

reindexes the order

Command

python

publish("order.placed", {"order_id": order.id, "total": str(order.total)})

Publish facts in the past tense (`order.placed`), not commands — a command puts every consumer's decision back in the producer. A queue delivers each message to one consumer (work distribution); a topic delivers to every subscriber (fan-out), and the mature shape is one topic feeding several subscriptions that are each a queue for one worker pool. Choosing wrongly in the fan-out direction is the dangerous one, because every message *is* processed and nothing looks broken while three of four systems never run. Django signals are synchronous and in-process — decoupling within a project, never an event bus.

2.Ordering, at-least-once delivery, and consumers that can be run twice

Brokers give you **at-least-once** delivery: a message is delivered until it is acknowledged, so a consumer that crashes after doing the work but before acknowledging will see it again. Ordering is guaranteed only within a partition, and only if you gave the broker a key to partition by. So the two properties a consumer needs are that processing the same event twice is harmless, and that arriving out of order does not corrupt state. Retries and a dead-letter queue handle the messages that keep failing.

Key features

Unacknowledged deliveries are "automatically requeued when the channel … is closed"
"Consumers must be prepared to handle redeliveries and … be implemented with idempotence in mind"
Deduplicate on `event_id` with a `UniqueConstraint` — concurrent deliveries race a `.exists()` check
Ordering holds within a partition only, and only if you set the partition key
A retrying poison message blocks everything behind it in an ordered partition
A DLQ needs an alert and a replay path, or it is just a slower way to lose data

The three delivery semantics, and why one of them is not on offer

The three delivery semantics, and why one of them is not on offer
Semantics	Mechanism	You get
at-most-once	ack before processing	loss on a crash — rarely acceptable
at-least-once	ack after processing	duplicates on a crash — the default
exactly-once delivery	—	not achievable: the ack can be lost
exactly-once processing	at-least-once + dedupe	what people actually mean, and it is your job

Retry classification, and where the message ends up

Retry classification, and where the message ends up
Failure	Retry	Then
downstream timeout / 5xx	yes, backoff + jitter	DLQ after N attempts
row not visible yet	yes, short delay	usually resolves on attempt 2
schema field missing	no	straight to the DLQ — retrying is deterministic failure
unparseable message	no	DLQ, and alert: something upstream is broken
already processed	no	acknowledge and stop — not an error
exhausted attempts	—	DLQ + alert + a replay path

One message, and every exit from a consumer

duplicatedeliveryout of ordernew, andcurrentsuccessprocess diesredelivered — the deduperow makes this safetransientfailureattempt n+1attemptsexhaustedpermanent failure— do not retry

delivered (at-least-once)

start

INSERT ProcessedEvent(event_id)

IntegrityError — already handled ack and stop

end

rank check: moves backwards? ack and stop

end

do the work

acknowledged — done

end

crash after work, before ack

retry with backoff + jitter

dead-letter queue + alert + replay path

end

Command

python

UniqueConstraint(fields=["consumer", "event_id"], name="uniq_processed_event")

At-least-once is the honest default, because the acknowledgement itself can be lost — so "exactly-once" is a *processing* property you provide, by deduplicating on the event id with a `UniqueConstraint`. Claim the event before doing the work: the reverse leaves a window in which redelivery repeats it. Ordering holds only within a partition and only if you set the key, so tolerate out-of-order arrival with a monotonic rank check rather than trusting it. Retry transient failures with backoff and jitter, dead-letter deterministic ones immediately — a retrying poison message blocks everything behind it — and give the DLQ an alert and a replay command.

3.Eventual consistency, the outbox, and versioning an event

Writing to your database and publishing to a broker are two systems, and there is no transaction spanning both — so a crash between them either loses the event or announces a change that rolled back. The **outbox pattern** removes the gap: write the event as a row in the *same transaction* as the change, and let a separate relay publish it afterwards. Consumers then catch up shortly after, which is **eventual consistency** — a window where the systems legitimately disagree. **Versioning** is what lets the event shape change without breaking them.

Key features

Dual writes have no transaction: save-then-publish loses events, publish-then-save invents them
The outbox makes the event a row in the same transaction — they commit or roll back together
A relay publishes committed rows and may republish; consumers already tolerate duplicates
Scale the relay with `select_for_update(skip_locked=True)` so several processes can share it
Carry a `version` from the first event ever published — adding one later is itself a break
A semantic change with no schema change (tax now included in `total`) is the most dangerous kind

Why the dual write cannot be made safe by reordering

Why the dual write cannot be made safe by reordering
Order of operations	Crash in between gives
`save()` then `publish()`	the change happened; no consumer ever hears — silent divergence
`publish()` then `save()`	consumers act on a change that never happened
`publish()` inside `atomic()`	the same, plus a published event for a rolled-back transaction
outbox row inside `atomic()`	nothing lost: the row and the change share one commit

Which event changes are safe

Which event changes are safe
Change	Safe?	How to do it
add an optional field	yes	consumers ignore what they do not read
add a required field	no	new version, or default it
rename a field	no	add the new name, publish both, remove later
remove a field	no	stop reading it, prove nobody does, then remove
tighten a type (`str` → `int`)	no	new version
change a field's meaning	no — and nothing detects it	a new field name, never a redefinition

The dual write, and the outbox that removes it

Above: two systems, no shared transaction — whichever order you choose, a crash in the gap produces an inconsistency. Below: the event is a row in your own database, so it commits with the change, and a relay publishes it afterwards.

Command

python

OutboxMessage.objects.create(topic="order.placed", payload=…)   # inside atomic()

You cannot write to your database and a broker atomically, and reordering only changes which failure you get — so make the event a row in the same transaction and let a relay publish it afterwards. Republishing is the safe direction, because consumers already deduplicate. Scale the relay with `select_for_update(skip_locked=True)`, and use the aggregate id as the partition key so per-aggregate order survives. Accept the lag as eventual consistency and model it in the product where a user can see it. Carry a `version` from your very first event, add fields rather than changing them, and never redefine a field's meaning — that is the break nothing detects.

Spotted a mistake on this page? Tell us.

Read Sheet