beginnerintermediateadvanced

System Design · Saga Pattern

Saga Pattern

Two coordination styles, and making compensating actions actually reliable

Choreographyservices react to each other's events
Orchestrationone coordinator directs every step
Compensating actionspurpose-built, per step — no automatic undo
Production realityidempotent, retried, and monitored for stuck sagas

1.Choreography: services react to events, no central coordinator

In a choreographed saga, there is no single component directing the workflow — each service completes its own local transaction and publishes a domain event describing what happened, and other services subscribe to those events and react by running their own local transactions in turn, publishing their own events afterward. The overall workflow emerges from this chain of event reactions rather than being explicitly controlled by any one place; no service needs to know the full sequence of steps, only which events it should react to and which event it should publish when it is done.

Key features

Each service completes its own local transaction, then publishes a domain event (e.g. "OrderPlaced," "PaymentCharged") describing what just happened
Other services subscribe to the events relevant to them and react by running their own local transaction, publishing their own resulting event when done — no service calls another directly to trigger the next step
No single service holds the full sequence of the workflow — each one only knows its own trigger events and its own resulting event, which keeps individual services simpler and more decoupled from the overall flow
The trade-off: with the workflow logic spread across every participating service instead of centralized in one place, understanding or debugging the full end-to-end flow requires tracing events across multiple services' logs rather than reading one orchestrator's code
Choreography fits naturally on top of an existing event-driven architecture (a message broker or event log already in use), since it is fundamentally just services reacting to each other's events — the same infrastructure already covered in this project's messaging sections

What each service knows and does in a choreographed saga

What each service knows and does in a choreographed saga
Service	Listens for	Does	Publishes
Order service	(starts the flow)	Creates the order	"OrderPlaced"
Payments service	"OrderPlaced"	Charges the customer	"PaymentCharged" or "PaymentFailed"
Inventory service	"PaymentCharged"	Reserves the item	"InventoryReserved" or "InventoryUnavailable"
Shipping service	"InventoryReserved"	Schedules a shipment	"ShipmentScheduled"

Choreography: each service reacts to the last event

OrderPlacedPaymentChargedInventoryReserved

Order service

creates the order

Payments service

charges the card

Inventory service

reserves the item

Shipping service

schedules shipment

Command

text

Order service:     commits, publishes OrderPlaced
Payments service:   subscribes OrderPlaced
                     commits, publishes PaymentCharged
Inventory service:  subscribes PaymentCharged
                     commits, publishes InventoryReserved
Shipping service:   subscribes InventoryReserved
                     commits, publishes ShipmentScheduled

Choreography: each service commits its own local transaction and publishes an event; other services react by subscribing to relevant events and running their own local transactions in turn. No service holds the full workflow — it emerges from the chain of reactions, which keeps individual services decoupled but makes the end-to-end flow harder to see in one place.

2.Orchestration: one coordinator controls the workflow

In an orchestrated saga, a single dedicated component — the orchestrator — explicitly directs the workflow: it tells each participating service what local transaction to execute next, waits for the result, and decides what to do based on that result (proceed to the next step, or trigger compensating actions if something failed). Unlike choreography, where the workflow emerges implicitly from services reacting to each other's events, orchestration makes the entire sequence explicit in one place — the orchestrator's own logic is the single source of truth for what the saga does at every step.

Key features

The orchestrator explicitly calls each participating service, telling it which local transaction to execute, and receives that service's result directly rather than reacting to a broadcast event
The orchestrator holds the full sequence of the workflow as its own logic — "call Payments, then call Inventory, then call Shipping, and if any step fails, call the compensating actions for the steps that already succeeded, in reverse order"
Because the whole flow lives in one place, orchestration is generally easier to understand, debug and monitor than choreography — a single orchestrator's state and logs show exactly where a given saga instance currently stands
The trade-off: the orchestrator becomes a more central, more tightly-coupled component — participating services now depend on the orchestrator calling them correctly, and the orchestrator itself needs its own reliability story (what happens if the orchestrator itself crashes mid-saga)
Orchestration is often implemented with a dedicated workflow/state-machine engine (e.g. AWS Step Functions, Temporal, Camunda) rather than hand-rolled coordination logic, specifically because reliably tracking "where is this saga instance right now" across possibly-long-running steps is itself a non-trivial problem

Choreography vs orchestration, the same underlying trade-off

Choreography vs orchestration, the same underlying trade-off
Property	Choreography	Orchestration
Where the workflow logic lives	Spread across every participating service	Centralized in one orchestrator
Coupling	Looser — services only know their own events	Tighter — services take direction from the orchestrator
Debugging the full flow	Requires tracing events across multiple services	One place shows the entire sequence and current state
Single point of workflow control	None — genuinely decentralized	The orchestrator — its own reliability now matters

Orchestration: one coordinator directs every step

Called directly, in sequence

Payments.charge

Inventory.reserve

Shipping.schedule

Command

text

orchestrator.run(order):
    charge_result = call(Payments, "charge", order)
    if not charge_result.ok: return fail(order)

    reserve_result = call(Inventory, "reserve", order)
    if not reserve_result.ok:
        call(Payments, "refund", order)   # compensate
        return fail(order)

    call(Shipping, "schedule", order)

Orchestration: a single coordinator explicitly calls each service, waits for the result, and decides the next step or the compensating actions directly — centralizing the workflow logic makes it easier to understand and debug than choreography, at the cost of tighter coupling and needing its own durable-state reliability story.

3.Compensating actions per step, and the failure-handling reality of running a saga

Every step in a saga that can commit real, externally-visible effects needs its own explicit compensating action — a separate operation designed specifically to semantically undo that step, since there is no database-level rollback spanning the whole saga. Making this reliable in practice means treating four things as first-class concerns, not afterthoughts: failure handling (deciding what triggers compensation and in what order), idempotency (a compensating action itself might be triggered more than once, so it needs to be safe to run twice), retries (a compensating action can itself fail transiently and needs its own retry policy), and partial completion (a saga can fail partway through even its own compensation sequence, leaving the system in a state that needs to be detected and recovered, not silently ignored).

Key features

A compensating action is designed per step, specifically for that step's real effect — "refund the charge" for "charge the card," "release the reservation" for "reserve the inventory" — not a generic, automatic inverse
Failure handling: a saga needs an explicit rule for what triggers compensation (a step failing, a timeout, an explicit cancellation) and in what order compensating actions run (typically reverse order of the original steps, undoing the most recent first)
Idempotency: a compensating action can be triggered more than once (a retry after an ambiguous timeout, a duplicate event in choreography) — it needs to produce the same end state whether it runs once or several times, using the same idempotency mechanisms (idempotency keys, unique constraints) already covered for message delivery in general
Retries: a compensating action is itself a real operation that can fail transiently (a network blip calling the refund API) — it needs its own bounded retry policy with backoff, the same discipline already covered for any other unreliable call
Partial completion: a saga can fail partway through running its OWN compensating actions (e.g. the refund succeeds but releasing inventory then fails) — this needs to be detected (monitoring, alerting, a stuck-saga reconciliation job) and resolved, not silently left in an inconsistent state nobody is watching

The four operational concerns, and why each is necessary

The four operational concerns, and why each is necessary
Concern	What happens without it	Standard fix
Failure handling	No clear rule for when/how compensation triggers or in what order	Explicit trigger conditions; reverse-order compensation
Idempotency	A duplicate compensation trigger double-refunds or double-releases	Idempotency key / unique constraint on the compensating action itself
Retries	A transient failure in compensation is treated as final, leaving the step un-compensated	Bounded retry with backoff, same as any other unreliable call
Partial completion	A saga stuck mid-compensation goes unnoticed indefinitely	Monitoring, alerting, and a reconciliation process for stuck sagas

Compensation, reverse order, itself fails partway

compensatethenstuck,detected

schedule() fails

triggers compensation

release_reservation()

succeeds

refund_charge()

fails — briefly unreachable

Reconciliation job

retries, then alerts

Command

text

# compensating actions run in REVERSE order of the
# original steps that already succeeded
original:    charge -> reserve -> schedule (fails here)
compensate:  release_reservation -> refund_charge
             (reverse order: undo the most recent
              successful step first)

Every saga step that commits a real effect needs an explicit, purpose-built compensating action, run in reverse order on failure. Making this reliable requires the same discipline as any other unreliable operation: idempotency (safe to trigger twice), bounded retries with backoff, and monitoring for sagas stuck mid-compensation — a saga can fail even while cleaning up after its own earlier failure, and that has to be detectable, not silently ignored.

Spotted a mistake on this page? Tell us.

Read Sheet