beginnerintermediateadvanced

Django · Section 87

State Machines and Workflows

Level: advanced
Read: 34 min
Concepts: 3

The section opens by drawing two machines — a payment moving `pending → processing → succeeded` or `→ failed`, an order moving `draft → placed → processing → fulfilled` or `→ cancelled` — and what makes them machines rather than word lists is the edges that are *absent*. `succeeded → pending` is not drawn; neither is `fulfilled → draft`. A `status` field with `choices` constrains the legal values and says nothing at all about legal moves, so the transition table has to be written somewhere explicit. Put it on the model as data, add a `CheckConstraint` because `choices` is validated by forms and cannot stop a `QuerySet.update()` or a migration writing nonsense, and make `transition_to()` the only place the field is ever assigned — that last discipline is what turns a grep for `.status =` into a genuine audit and gives every future requirement one place to attach to. Distinguish two rejections while you are there: a request to move to the state the object is already in is a duplicate — a retried task, a double-clicked button — and should succeed quietly, while a genuinely unreachable move is a 409, because the caller's view of the world is stale. The moment two workers exist, the read-check-write sequence becomes a race, and `select_for_update()` closes it only if the same statement both locks and reads: a lock taken over an object loaded earlier checks a stale value and prevents nothing. Django refuses to run it outside `atomic()` at all. Keep the transaction short and deliberate — slow and remote work happens before it, because a transaction wrapping a payment call holds its row lock for the provider's p99 and converts their latency into your contention. Inside the boundary goes everything that must be consistent with the state change: the status, the history row, the stock movement. Outside it, through `on_commit`, goes everything irreversible: emails, queue messages, published events. The last concept is what the transition leaves behind. Idempotency mostly comes free — after the move, the same request is a no-op, so the state *is* the key and explicit keys are needed only at the edges. Events must be emitted from inside the transition and after the commit, or they either describe a rollback or miss a second code path; where they must never be lost, they become an outbox row written in the same transaction. And history is domain data, not just audit: a row per transition carrying both the from-state and the to-state, which makes the `status` field a cache of the latest change rather than the only record of what happened.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

The machine is the set of edges; choices only constrains the vertices.
One method writes the status field, and a CheckConstraint backs the value set.
Lock and read in one statement, then re-check — otherwise the lock does nothing.
Irreversible work belongs after the commit; remote work belongs before the transaction.
The state check gives idempotency; the transaction makes event and state agree.

What you will be able to do

Write a workflow whose illegal moves cannot be performed by any code path
Stop two workers from processing the same order twice
Emit exactly one event per transition, never for a rollback
Answer "how long was this in processing?" from data rather than by guessing

One transition request, and every gate it passes

POST /orders/88/process

a view, a task retry, or an admin action

Remote work first

reserve stock, call the provider — outside any transaction

BEGIN

the boundary opens here, not earlier

select_for_update().get()

lock AND read in one statement

Already in the target state?

duplicate request — return quietly, no error

Is the edge in the table?

placed → processing: yes · fulfilled → draft: no

409 Conflict

the caller’s view of the world is stale

UPDATE status + StatusChange + OutboxMessage

one transaction: state, history and event agree or none exist

COMMIT

lock released; on_commit hooks fire now

Email · queue · publish

irreversible work, only after durability

The table, and the moves that are not in it

Making the workflow explicit, and enforcing the value set where `choices` cannot.

Valid transitions — and making the invalid ones impossible

coreintermediate

The roadmap draws two machines: a payment goes `pending → processing → succeeded`, or `→ failed`; an order goes `draft → placed → processing → fulfilled`, or `→ cancelled`. What makes that a state machine rather than a list of words is the set of transitions that are **not** drawn — `succeeded → pending` is absent, and so is `fulfilled → draft`. A `status` field with `choices` enforces the legal *values* and says nothing about legal *moves*, so the transition table has to be written down and checked somewhere.

Think of it as

A status field is usually introduced as a label and then quietly becomes the most important invariant in the system, because money, emails and shipments all hang off it. The shift worth making is to stop thinking about which values exist and start thinking about which *edges* exist. Once the edges are explicit, three useful things follow. First, illegal moves become detectable: `refunded → succeeded` is not a bug you have to reason about, it is an edge that is not in the table. Second, the shape of the workflow becomes reviewable — you can look at the map and ask whether a cancelled order should really be able to become placed again, which is a product question that otherwise gets decided by whichever view happened to be written first. Third, the terminal states become visible, and terminal states are what keep a workflow from cycling forever. Where to enforce it matters as much as writing it down. `choices` is validated by forms and `full_clean()`, not by the database, so it cannot stop a `QuerySet.update()` or a data migration writing nonsense — a `CheckConstraint` can, and belongs there for the value set. The transition rule itself is application logic, and it belongs in one method on the model rather than scattered across views, tasks and admin actions, because a rule with five copies has five chances to disagree. The single most valuable habit is to make the transition the only way the field changes: no `order.status = "cancelled"` anywhere except inside `transition_to`. And when a transition is rejected, distinguish two cases in how you respond. A move to the state the object is *already* in is usually a duplicate request — a double-clicked button, a retried webhook — and should be a quiet success, not an error. A move to a genuinely unreachable state is a real conflict and deserves a 409, because it means the caller's idea of the world is out of date.

python

order.transition_to(Status.PLACED, actor=request.user)   # the only way status changes

What we're doing: Put the transition table on the model, enforce the value set in the database, and make one method the only path that writes the field.

orders/models.pypython

class Order(models.Model):
    class Status(models.TextChoices):
        DRAFT = "draft", "Draft"
        PLACED = "placed", "Placed"
        PROCESSING = "processing", "Processing"
        FULFILLED = "fulfilled", "Fulfilled"
        CANCELLED = "cancelled", "Cancelled"

    status = models.CharField(
        max_length=12, choices=Status.choices, default=Status.DRAFT, db_index=True
    )

    # The edges, as data. Absent edges are the whole point: "fulfilled" and
    # "cancelled" map to empty sets, which is what makes them terminal.
    TRANSITIONS = {
        Status.DRAFT: {Status.PLACED, Status.CANCELLED},
        Status.PLACED: {Status.PROCESSING, Status.CANCELLED},
        Status.PROCESSING: {Status.FULFILLED, Status.CANCELLED},
        Status.FULFILLED: set(),
        Status.CANCELLED: set(),
    }

    class Meta:
        constraints = [
            # choices is validated by forms, not by the database. This is what
            # stops a data migration or an update() writing "shipped".
            models.CheckConstraint(
                condition=models.Q(status__in=[c[0] for c in Status.choices]),
                name="order_status_valid",
            ),
        ]

    def can_transition_to(self, target):
        return target in self.TRANSITIONS[self.status]

    def transition_to(self, target, *, actor=None):
        """The ONLY place Order.status is assigned. Everything else calls
        this, so the rule cannot be bypassed by a view written in a hurry."""
        if self.status == target:
            # A duplicate request — a double-clicked button, a retried task.
            # Not an error: the caller's desired outcome already holds.
            return False

        if not self.can_transition_to(target):
            raise InvalidTransition(
                f"Order {self.pk} cannot move {self.status} → {target}"
            )

        self.status = target
        self.save(update_fields=["status", "updated_at"])
        return True

15–21: The table is the specification. Reading it answers product questions — can a cancelled order be placed again? — that would otherwise be settled implicitly by whichever view was written first.
27–30: `choices` alone is validated by forms and `full_clean()`, so nothing stops `update(status="shipped")` or a migration writing a typo. The check constraint is the database's copy of that rule.
38–41: Re-entering the current state returns `False` rather than raising. Duplicate requests are the normal case with retried tasks and impatient users, and treating them as errors produces alerts about a system working correctly.
43–46: A genuinely illegal move raises with both states named. "Invalid transition" with no values is the error message you will be reading in six months.
48–49: `update_fields` keeps the write narrow, which matters when the next concept adds locking — a narrow update holds its row lock for less time.

Why this works: Illegal moves raise with a readable message, duplicate requests succeed quietly, the database rejects values no code path should produce, and the workflow is legible from one table.

Assigning the status field directly

Wrong

python

# in a view
order.status = "cancelled"
order.save()
# in a task, three months later
order.status = "processing"      # from cancelled. Nothing stops it.

Better

python

order.transition_to(Order.Status.CANCELLED, actor=request.user)

What you see: Orders appear in states the workflow diagram says are unreachable — cancelled orders being fulfilled, refunded payments back in `processing` — and no single commit looks wrong.

Why: A transition rule enforced in one method and bypassed everywhere else is not enforced. Direct assignment is easy to write, invisible in review (a two-line change in a view), and each instance individually looks reasonable — the damage comes from their combination across code paths written months apart. Making `transition_to()` the only writer means the rule lives in one place, and a grep for `\.status =` becomes a genuine audit. It also gives every future requirement — an event, an audit row, a notification — one place to hang from.

The order machine — and the moves that are absent on purpose

draft

start

placed

processing

fulfilled

end

cancelled

end

The roadmap's two machines, written as a transition table

The roadmap's two machines, written as a transition table
Object	From	May move to
Payment	`pending`	`processing`
Payment	`processing`	`succeeded`, `failed`
Payment	`succeeded`	— (terminal, except a modelled refund)
Payment	`failed`	— (terminal; a retry is a new payment)
Order	`draft`	`placed`, `cancelled`
Order	`placed`	`processing`, `cancelled`
Order	`processing`	`fulfilled`, `cancelled`
Order	`fulfilled`	— (terminal)

Together

python

TRANSITIONS = {
    "draft": {"placed", "cancelled"},
    "placed": {"processing", "cancelled"},
    "processing": {"fulfilled", "cancelled"},
    "fulfilled": set(),
    "cancelled": set(),
}

Where each rule can actually be enforced

Where each rule can actually be enforced
Rule	Enforced by	Survives `QuerySet.update()`?
the value is one of the choices	`choices` + forms/`full_clean()`	no
the value is one of the choices	`CheckConstraint`	yes — the database rejects it
the move is legal	`transition_to()` in application code	no — so nothing else may assign
no two workers move it at once	`select_for_update()`	n/a — see the next concept
a terminal state stays terminal	an empty set in the table	no

Together

python

constraints = [
    models.CheckConstraint(
        condition=models.Q(status__in=[c[0] for c in Status.choices]),
        name="order_status_valid",
    ),
]

Remember: A `status` field with `choices` constrains values, never moves — the machine is the set of *edges*, and the edges that are absent (`fulfilled → draft`) are the ones doing the work. Write the table as data on the model, add a `CheckConstraint` so the database enforces the value set that `choices` only validates in forms, and make `transition_to()` the single place the field is ever assigned so a grep is a real audit. Re-entering the current state is a duplicate request and should succeed quietly; a genuinely unreachable move is a 409.

Two workers, one row

Locking that actually locks, and deciding what belongs inside the transaction.

Two workers, one order: locking and the transaction boundary

coreadvanced

Checking a state and then changing it is two statements, and anything can happen between them. Two workers can both read `placed`, both decide the move to `processing` is legal, and both perform it — shipping twice, or charging twice. `select_for_update()` closes the gap by locking the row until the transaction ends, so the second worker waits and then re-reads the *new* state. The matching decision is the transaction boundary: what belongs inside it, and what must be deferred until after it commits.

Think of it as

The read-check-write sequence is the whole problem, and it is invisible in single-threaded testing. Under concurrency the window between the read and the write is where a second actor slips in, and the fix is to make the read itself exclusive: `select_for_update()` takes a row lock that other transactions wanting the same row must wait for. Crucially, the value of the lock comes from what happens after the wait — the second worker does not simply resume, it re-reads the row and now sees `processing`, so its own transition check fails and it does nothing. That means the lock and the transition check must be inside the *same* transaction; a lock released before the check is not a lock. Django enforces part of this for you by refusing to run `select_for_update()` outside a transaction at all. Once locking is right, the second question is what else belongs inside the boundary, and the rule is short: everything that must be consistent with the state change goes inside, and everything that cannot be undone goes outside. Database writes that form one logical change — the status, the audit row, the domain event — go inside, because a status change without its event is precisely the inconsistency the transaction exists to prevent. Anything that leaves the process must not: an email, an HTTP call to a payment provider, a queue message. Those cannot be rolled back, so performing one inside a transaction that later fails means telling the outside world about something that did not happen. `transaction.on_commit()` is the mechanism, and it also fixes the subtler ordering bug — a task enqueued mid-transaction can be picked up by a worker before the row is visible. The third consideration is how long the lock is held, because a held lock is a queue everyone else stands in. Do the slow work — rendering, calling the provider, computing — before opening the transaction, then lock, re-check, write and commit quickly. A transaction that wraps an HTTP call holds its lock for the remote service's p99, which converts a slow dependency into database contention.

python

with transaction.atomic():
    order = Order.objects.select_for_update().get(pk=pk)   # lock, then re-check

What we're doing: Move an order to `processing` exactly once under concurrency, keeping the remote call outside the lock and every irreversible side effect after the commit.

orders/services.pypython

def begin_processing(order_id, *, actor=None):
    # 1. SLOW WORK FIRST, outside any transaction. A provider call inside
    #    the block below would hold the row lock for their p99, turning a
    #    slow dependency into database contention for everyone.
    reservation = warehouse.reserve(order_id)      # HTTP, ~300 ms

    with transaction.atomic():
        # 2. Lock and RE-READ. The re-read is the point: a worker that
        #    waited here now sees whatever the winner wrote.
        order = Order.objects.select_for_update().get(pk=order_id)

        if not order.can_transition_to(Order.Status.PROCESSING):
            # Already processing, or cancelled while we were reserving stock.
            warehouse.release(reservation)         # compensate the slow work
            return False

        order.status = Order.Status.PROCESSING
        order.save(update_fields=["status", "updated_at"])

        # 3. Everything that must be consistent with the status goes INSIDE.
        OrderEvent.objects.create(order=order, kind="processing_started",
                                  actor=actor, reservation=reservation.id)
        Stock.objects.filter(sku__in=order.skus()).update(
            reserved=F("reserved") + 1
        )

        # 4. Everything irreversible goes AFTER the commit. An email sent
        #    inside a transaction that later rolls back cannot be recalled,
        #    and a task enqueued here can outrun its own row.
        transaction.on_commit(lambda: send_processing_email.delay(order.id))
        transaction.on_commit(lambda: publish_event.delay("order.processing", order.id))

    return True

2–5: The ordering that matters most for throughput. Holding a row lock across an HTTP call means every other worker wanting that order waits on the warehouse API, not on your database.
10: `select_for_update()` outside `atomic()` raises `TransactionManagementError`, so Django will not let you take a lock that would be released immediately.
12–15: The re-check after acquiring the lock. Without it the lock is pointless — the loser would proceed on the state it read before waiting. Note the compensation: the reservation made outside the transaction has to be released by hand.
21–25: The event and the stock change share the transaction with the status change, because a status with no event, or a reservation with no stock movement, is exactly the inconsistency the boundary exists to prevent.
30–31: Both side effects deferred. `on_commit` fixes two problems at once: an email cannot be un-sent after a rollback, and a task enqueued mid-transaction can be picked up before the row it names is visible.

Why this works: Only one worker performs the transition, the loser withdraws cleanly, the database work is atomic, no lock is held across a network call, and nothing leaves the process until the change is durable.

Locking but not re-reading

Wrong

python

order = Order.objects.get(pk=pk)                 # read BEFORE the lock
with transaction.atomic():
    Order.objects.select_for_update().filter(pk=pk).exists()   # "lock"
    order.transition_to(Order.Status.PROCESSING)  # checks the STALE object

Better

python

with transaction.atomic():
    order = Order.objects.select_for_update().get(pk=pk)   # lock AND read
    order.transition_to(Order.Status.PROCESSING)

What you see: The double-processing bug survives the addition of locking, and now looks impossible — the code visibly takes a lock, so the race is assumed to be elsewhere.

Why: A lock does not retroactively refresh objects you already loaded. If the instance was read before the lock was taken, its `status` is whatever it was then, so the transition check runs against a stale value and the second worker proceeds exactly as it did without any locking. The lock has to be the same statement that produces the object you check, which is why `select_for_update().get(...)` is the idiom rather than a separate locking call.

Two workers take the same order — with the lock, and what the loser does

Worker A

Worker B

Database

1. BEGIN; SELECT … FOR UPDATE (order 88)the row lock is taken here
2. status = "placed"
3. BEGIN; SELECT … FOR UPDATE (order 88)blocks — A holds the lock
4. check: placed → processing is legal
5. UPDATE status = "processing"; INSERT event
6. COMMITlock released; on_commit hooks now fire
7. status = "processing"B re-reads and sees the NEW value — this is the point of the lock
8. check: processing → processing is a no-opreturns quietly; the order ships once

Pessimistic or optimistic — both work, differently

Pessimistic or optimistic — both work, differently
Approach	How it prevents the race	Best when
`select_for_update()`	the second reader waits for the lock	the transition does real work while holding it
conditional `update()`	the `WHERE` clause fails for the loser	the transition is a single field change
`skip_locked=True`	the second worker takes another row	a worker pool draining a queue table
`nowait=True`	the second worker fails immediately	you would rather error than queue
a database version column	the update matches on the old version	long user-facing edits ("someone else changed this")

Together

python

# Optimistic: the check IS the write. updated == 0 means somebody won first.
updated = Order.objects.filter(pk=order.pk, status="placed").update(status="processing")
if not updated:
    return                       # another worker already moved it

Inside the transaction, or after it commits

Inside the transaction, or after it commits
Action	Where	Why
the status change	inside	it is the change
the audit row / domain event	inside	a change with no record is the gap to avoid
stock decrement, balance update	inside	must be consistent with the state
sending an email	`on_commit`	cannot be un-sent
enqueuing a task	`on_commit`	a worker can outrun the commit, or survive a rollback
calling a payment provider	before the transaction	never hold a row lock for a remote p99

Together

python

with transaction.atomic():
    ...                                        # writes only
    transaction.on_commit(lambda: notify.delay(order.id))

Remember: Read-check-write is a race. `select_for_update()` closes it, but only if the *same statement* both locks and reads — a lock over an object you loaded earlier checks a stale value and changes nothing. It must be inside `atomic()`; Django raises otherwise. Keep the transaction short: do slow and remote work before it, hold the lock only for the check and the writes. Everything that must be consistent with the state change goes inside the boundary; everything irreversible — email, queue messages, provider calls — goes in `on_commit` or before it. And `skip_locked=True` is the right tool when a pool of workers is draining rows.

What a transition leaves behind

Idempotency for free, events that match reality, and a history the product can query.

Idempotency, events, and the history the transition leaves behind

standardadvanced

Three things hang off a transition. **Idempotency**: the same request arriving twice must produce one transition, which the state check itself gives you for free — the second attempt finds the object already moved and does nothing. **Event generation**: a transition is the natural place to emit "order.fulfilled" for other parts of the system, and it must be emitted exactly as often as the transition happens. **Audit history**: a row per transition, so the object's timeline is queryable rather than inferred from a single current-status field.

Think of it as

The state machine turns out to be an idempotency mechanism you already built. A transition guarded by "is this move legal from where I am now?" is naturally at-most-once, because after the first success the object is no longer in the state the move starts from — so the second attempt is either a no-op or an illegal move. That is a stronger property than it sounds, and it means the usual advice about idempotency keys applies mainly at the *edges*, where a request arrives before any state has changed. Inside the workflow, the state is the key. Events are where the discipline is easy to lose. The requirement is that an event is emitted exactly when the transition actually occurred, and the two ways to break it are symmetrical: publishing before the commit means an event for something that may roll back, and publishing outside the transition — in a view, say — means a second code path that moves the same object emits nothing. Emitting inside `transition_to`, through `on_commit`, fixes both: one place, and only after the change is durable. If subscribers must never miss an event even when the broker is down, the event becomes a row written in the same transaction and a relay publishes it afterwards — the outbox pattern, which the event-driven section develops. History is the third, and it is worth separating from the audit log even though they overlap. An audit row answers "who did this" for compliance; a transition row is *domain* data — the timeline a customer support agent reads, the timestamps a "delivered within 48 hours" SLA is computed from, the data behind "how long do orders sit in processing?". Because it is domain data it belongs in an ordinary model with the previous state, the new state, the actor, the reason, and the time. Storing both states rather than just the new one is what makes the history self-contained: you can read one row and know the move, without reconstructing it from the row before. And once that table exists, the current `status` field becomes a cache of the last transition — worth keeping for query performance, but no longer the only record of what happened.

python

transaction.on_commit(lambda: publish("order.fulfilled", order.id))

What we're doing: One `transition_to` that is idempotent, writes history, and emits an event exactly once — with an outbox row for the events subscribers must not miss.

orders/models.pypython

def transition_to(self, target, *, actor=None, reason=""):
    """The single writer of Order.status — and therefore the single place
    history and events are produced."""
    previous = self.status

    if previous == target:
        return False                       # duplicate request: already done

    if target not in self.TRANSITIONS[previous]:
        raise InvalidTransition(f"Order {self.pk}: {previous} → {target}")

    with transaction.atomic():
        # The conditional update IS the concurrency check: if another worker
        # moved the row first, WHERE status = previous matches nothing.
        updated = (
            Order.objects.filter(pk=self.pk, status=previous)
            .update(status=target, updated_at=timezone.now())
        )
        if not updated:
            return False                   # somebody else won the race

        self.status = target

        # Domain history: from AND to, so the row stands alone.
        StatusChange.objects.create(
            order=self, from_state=previous, to_state=target,
            actor=actor, reason=reason, occurred_at=timezone.now(),
        )

        # An outbox row, written in the SAME transaction as the change.
        # If the broker is down the event is still recorded, and a relay
        # publishes it later — the event cannot be lost or invented.
        OutboxMessage.objects.create(
            topic=f"order.{target}",
            payload={"order_id": self.pk, "from": previous, "to": target},
        )

    # Fast-path publish AFTER the commit. The relay is the safety net; this
    # is only the low-latency path, and it must never run before the commit.
    transaction.on_commit(lambda: publish_outbox.delay())
    return True

6–7: The idempotency guard, and it is the state machine doing the work — no key, no cache, no dedup table. The object has already moved, so there is nothing left to do.
15–20: The conditional `update()` makes the check and the write one statement, so a concurrent transition is detected by a row count of zero rather than by a lock. This is the optimistic sibling of `select_for_update()`.
25–28: Both states on the row. Storing only `to_state` forces every reader to look at the preceding row to know what the move was, which breaks as soon as rows are filtered or paginated.
32–36: The outbox row shares the transaction with the status change, so the event and the state can never disagree — either both are committed or neither is.
38–40: `on_commit` for the fast path only. If this task never runs, the relay picks the row up anyway; if it ran *before* the commit, it would publish an event for a change that might roll back.

Why this works: A repeated request changes nothing, a concurrent one is detected without a lock, the history is self-describing, and the event is guaranteed to match the transition exactly — never published for a rollback, never lost to a broker outage.

One order, read from its history table rather than its status field

Mon 09:12
draft → placed
actor: the customer · reason: checkout completed
Mon 09:12
event: order.placed
emitted on_commit — after the row is durable
Mon 09:14
placed → processing
actor: system · reason: payment succeeded
Mon 09:14
placed → processing (rejected)
a retried task: already processing, so a quiet no-op
Tue 16:40
processing → fulfilled
actor: warehouse · reason: dispatched, tracking AB123
Tue 16:40
event: order.fulfilled
one event, because there was one transition
Fri 11:02
fulfilled → cancelled (rejected)
not an edge in the table — 409, and the caller is out of date

What each of the three gives you, and where it goes

What each of the three gives you, and where it goes
Concern	Mechanism	Where
idempotency inside the workflow	the transition check itself	`transition_to()`
idempotency at the edge	a client-supplied key	the view / webhook handler
events for other systems	emit on successful transition	`on_commit`, from `transition_to()`
events that must not be lost	an outbox row + a relay	inside the transaction
domain history	a `StatusChange` row	inside the transaction
compliance audit	an `AuditEntry` row	inside the transaction

Together

python

StatusChange.objects.create(
    order=order, from_state=previous, to_state=target,
    actor=actor, reason=reason, occurred_at=timezone.now(),
)

Remember: The transition check *is* your idempotency mechanism inside the workflow — after the move, the same request is a no-op, so keys are only needed at the edges. Emit events from inside `transition_to` and inside `on_commit`: emitting elsewhere means a second code path moves the object silently, and emitting before the commit means announcing a change that may roll back. Where an event must never be lost, write it as an outbox row in the same transaction and let a relay publish it. Store `from_state` and `to_state` on the history row so it stands alone — and treat `status` as a cache of the last transition rather than the record itself.