beginnerintermediateadvanced

System Design · Section 36

Saga Pattern

Level: advanced
Read: 22 min
Concepts: 3

A saga coordinates a sequence of local transactions across services one of two ways. Choreography has no central coordinator — each service commits its own step and publishes a domain event, and other services react by subscribing to relevant events and running their own steps in turn, keeping services decoupled at the cost of the overall workflow only existing implicitly across every participant. Orchestration centralizes control in a single coordinator that explicitly calls each service, waits for the result, and decides what happens next — easier to trace and debug, at the cost of tighter coupling to that one coordinator. Whichever style is chosen, every step that commits a real effect needs its own purpose-built compensating action (not an automatic inverse), and making that reliable in production requires the same discipline as any other unreliable operation: idempotency, bounded retries, and monitoring for sagas that get stuck even mid-compensation.

This section

Saga Design pattern Spring Boot Microservices Interview Questions and Answers | Code DecodeCode Decode

What is true here

Choreography: services react to each other's events, no central coordinator — decoupled, but the full workflow only exists implicitly.
Orchestration: one coordinator explicitly directs every step and every failure decision — centralized, easier to trace, more tightly coupled.
A compensating action is designed per step for that step's real effect — not every step necessarily has a clean inverse.
Compensation needs the same reliability discipline as any other operation: idempotency, bounded retries, and monitoring for a saga stuck mid-compensation.

What you will be able to do

Choose between choreography and orchestration based on a workflow's actual coordination complexity
Design a compensating action for a given saga step, including its idempotency and retry needs
Recognize and plan for a saga failing partway through its own compensation sequence

Two coordination styles

Choreography's decentralized event reactions versus orchestration's single explicit coordinator, and the trade-off between them.

Choreography: services react to events, no central coordinator

coreadvanced

In a choreographed saga, there is no single component directing the workflow — each service completes its own local transaction and publishes a domain event describing what happened, and other services subscribe to those events and react by running their own local transactions in turn, publishing their own events afterward. The overall workflow emerges from this chain of event reactions rather than being explicitly controlled by any one place; no service needs to know the full sequence of steps, only which events it should react to and which event it should publish when it is done.

Think of it as

A choreographed dance has no single director calling out every dancer's move in real time — each dancer has learned their own part and reacts to specific cues from the music or from what another dancer just did, and the overall performance emerges from all of those individual, decentralized reactions happening in the right sequence. No single dancer holds a master script of the whole performance; each one only needs to know their own cues and their own moves. A choreographed saga works the same way — each service only needs to know which events to listen for and which event to publish when its own step finishes, not the shape of the entire workflow.

text

Order service:     commits, publishes OrderPlaced
Payments service:   subscribes OrderPlaced
                     commits, publishes PaymentCharged
Inventory service:  subscribes PaymentCharged
                     commits, publishes InventoryReserved
Shipping service:   subscribes InventoryReserved
                     commits, publishes ShipmentScheduled

What we're doing: Trace an order-placement choreographed saga through 4 services reacting to each other's events, with no central coordinator.

choreographed-saga.txttext

1. Order service receives "place order" request.
   Commits its own local transaction (creates the
   order row). Publishes event: OrderPlaced.

2. Payments service, subscribed to OrderPlaced,
   receives the event. Charges the customer's card.
   Commits its own local transaction. Publishes
   event: PaymentCharged.

3. Inventory service, subscribed to PaymentCharged,
   receives the event. Reserves the item. Commits
   its own local transaction. Publishes event:
   InventoryReserved.

4. Shipping service, subscribed to InventoryReserved,
   receives the event. Schedules a shipment. Commits
   its own local transaction. Publishes event:
   ShipmentScheduled.

Notice: no service in this chain called any other
service directly, and no single service holds the
"full recipe" of all 4 steps -- each one only knows
its own one trigger event and its own one
resulting event.

3: This is the entire coordination mechanism — an event, not a direct call to the next service in line.
20: This is the defining property of choreography — the full workflow only exists as the emergent sum of 4 independent, locally-scoped reactions.

Why this works: This trace shows exactly what "no central coordinator" means concretely — every transition happens because a service reacted to an event it was already subscribed to, not because anything told it "now do step 3."

Losing track of the overall workflow because no single place documents it

Wrong

text

# 6 services, each independently subscribing to
# events from 2-3 others, with the full event
# graph existing only implicitly across 6
# separate codebases -- no diagram, no single
# source of truth for "what happens when an
# order is placed"

Better

text

# maintain an explicit event-flow diagram
# (even though no service enforces it at
# runtime) documenting which service publishes
# which event and which services subscribe to
# it, kept up to date as part of changing any
# step

What you see: A new engineer (or even an experienced one debugging an incident) cannot answer "what happens after an order is placed" without individually reading through 6 different services' event-subscription code, because choreography's decentralization means the full workflow was never written down anywhere as a single artifact.

Why: Choreography's core strength — no single service needs to know the whole flow — is also its core documentation risk, because nothing forces the whole flow to be written down anywhere either; without a deliberately maintained diagram or specification outside the code, the only way to reconstruct the full picture is to read every participating service.

Choreography: each service reacts to the last event

Order service

creates the order

Payments service

charges the card

Inventory service

reserves the item

Shipping service

schedules shipment

What each service knows and does in a choreographed saga

What each service knows and does in a choreographed saga
Service	Listens for	Does	Publishes
Order service	(starts the flow)	Creates the order	"OrderPlaced"
Payments service	"OrderPlaced"	Charges the customer	"PaymentCharged" or "PaymentFailed"
Inventory service	"PaymentCharged"	Reserves the item	"InventoryReserved" or "InventoryUnavailable"
Shipping service	"InventoryReserved"	Schedules a shipment	"ShipmentScheduled"

Remember: Choreography: each service commits its own local transaction and publishes an event; other services react by subscribing to relevant events and running their own local transactions in turn. No service holds the full workflow — it emerges from the chain of reactions, which keeps individual services decoupled but makes the end-to-end flow harder to see in one place.

Orchestration: one coordinator controls the workflow

coreadvanced

In an orchestrated saga, a single dedicated component — the orchestrator — explicitly directs the workflow: it tells each participating service what local transaction to execute next, waits for the result, and decides what to do based on that result (proceed to the next step, or trigger compensating actions if something failed). Unlike choreography, where the workflow emerges implicitly from services reacting to each other's events, orchestration makes the entire sequence explicit in one place — the orchestrator's own logic is the single source of truth for what the saga does at every step.

Think of it as

An orchestrated saga is a stage director calling out cues, rather than a choreographed dance where each performer reacts to their own trigger. The director watches everything, tells each actor exactly when to enter and what to do, and if something goes wrong (an actor misses their cue), the director is the one who decides how to recover — skip ahead, redo a scene, or stop the show. Every actor takes direction from the same single source, and the director's own script is the one place that shows the entire performance from start to finish, unlike a choreographed piece where no single performer has the full picture.

text

orchestrator.run(order):
    charge_result = call(Payments, "charge", order)
    if not charge_result.ok: return fail(order)

    reserve_result = call(Inventory, "reserve", order)
    if not reserve_result.ok:
        call(Payments, "refund", order)   # compensate
        return fail(order)

    call(Shipping, "schedule", order)

What we're doing: Trace the same order-placement saga as an orchestrated flow, contrasting directly with choreography's event-chain version.

orchestrated-saga.txttext

Order Orchestrator.run(order):

1. Orchestrator calls Payments.charge(order)
   directly. Waits for the result.
   -> Payments commits its own local transaction,
      returns success/failure to the orchestrator.

2. If successful, orchestrator calls
   Inventory.reserve(order) directly. Waits.
   -> Inventory commits its own local transaction,
      returns success/failure.

3. If that also succeeded, orchestrator calls
   Shipping.schedule(order) directly. Waits.

4. If step 3 fails, the ORCHESTRATOR (not
   Shipping, not Inventory) decides what happens
   next: it calls Inventory.release(order) and
   Payments.refund(order) itself, in that order,
   because its own logic is the single place that
   knows the full sequence and how to unwind it.

3: The orchestrator calls Payments directly and waits — this is the structural difference from choreography, where Payments would instead subscribe to an event.
15: This is the concrete payoff of centralization — recovering from a failure is a decision the orchestrator makes directly, using its own single view of what already succeeded, rather than something reconstructed from multiple services' independent event reactions.

Why this works: Comparing this trace directly against the choreography concept's trace of the same workflow makes the trade-off concrete — the same 4 steps, but here every transition and every failure decision passes through one place instead of emerging from independent event reactions.

Building the orchestrator without a plan for the orchestrator itself crashing mid-saga

Wrong

text

# orchestrator keeps saga progress only in its
# own in-memory process state
def run(order):
    step = "charging"
    charge(order)
    step = "reserving"   # if the process crashes
    reserve(order)         # right here, this saga
                             # instance's progress
                             # is gone entirely

Better

text

# orchestrator persists saga progress durably
# (a database row, or a workflow-engine's own
# durable state) after EACH step, so a crash
# can resume from the last completed step
# instead of losing track of the saga entirely
def run(order):
    persist_state(order.id, "charging")
    charge(order)
    persist_state(order.id, "reserving")
    reserve(order)

What you see: An order gets charged and its inventory reserved, then the orchestrator process crashes before scheduling shipping — and because the saga's progress only existed in that process's memory, nothing ever resumes it or runs the compensating actions, leaving the order permanently stuck in a half-completed state that nobody's logic is watching anymore.

Why: Centralizing the workflow logic in one orchestrator also centralizes the risk of that one component failing — an orchestrator that does not durably persist which step a given saga instance has reached cannot recover or resume correctly after a crash, which is exactly the kind of reliability problem dedicated workflow engines (Step Functions, Temporal) exist to solve for real production sagas.

Orchestration: one coordinator directs every step

Called directly, in sequence

Payments.charge

Inventory.reserve

Shipping.schedule

Choreography vs orchestration, the same underlying trade-off

Choreography vs orchestration, the same underlying trade-off
Property	Choreography	Orchestration
Where the workflow logic lives	Spread across every participating service	Centralized in one orchestrator
Coupling	Looser — services only know their own events	Tighter — services take direction from the orchestrator
Debugging the full flow	Requires tracing events across multiple services	One place shows the entire sequence and current state
Single point of workflow control	None — genuinely decentralized	The orchestrator — its own reliability now matters

Remember: Orchestration: a single coordinator explicitly calls each service, waits for the result, and decides the next step or the compensating actions directly — centralizing the workflow logic makes it easier to understand and debug than choreography, at the cost of tighter coupling and needing its own durable-state reliability story.

Making compensation reliable in production

Designing a purpose-built compensating action per step, and the operational discipline — idempotency, retries, monitoring — that keeps it working under real failure conditions.

Compensating actions per step, and the failure-handling reality of running a saga

coreadvanced

Every step in a saga that can commit real, externally-visible effects needs its own explicit compensating action — a separate operation designed specifically to semantically undo that step, since there is no database-level rollback spanning the whole saga. Making this reliable in practice means treating four things as first-class concerns, not afterthoughts: failure handling (deciding what triggers compensation and in what order), idempotency (a compensating action itself might be triggered more than once, so it needs to be safe to run twice), retries (a compensating action can itself fail transiently and needs its own retry policy), and partial completion (a saga can fail partway through even its own compensation sequence, leaving the system in a state that needs to be detected and recovered, not silently ignored).

Think of it as

Think of a saga's compensating actions like a fire escape plan for a building with several floors already occupied when the alarm sounds. Each floor needs its own specific evacuation procedure (a compensating action) — you cannot just "undo" people already being on floor 3, you have to actually walk them down. That evacuation plan has to work even if it gets triggered twice by mistake (idempotency — running it twice should not cause chaos), has to have a fallback if a stairwell is blocked on the first attempt (retries), and needs a way to notice and handle the case where the evacuation itself only gets halfway done before something else goes wrong (partial completion) — none of which is optional or an edge case, because a fire escape plan that only works when everything else also goes right is not actually a plan.

text

# compensating actions run in REVERSE order of the
# original steps that already succeeded
original:    charge -> reserve -> schedule (fails here)
compensate:  release_reservation -> refund_charge
             (reverse order: undo the most recent
              successful step first)

What we're doing: Show a saga's compensation sequence itself failing partway through, and the reconciliation process that catches it.

partial-compensation-failure.txttext

Saga: charge -> reserve -> schedule (fails)

Compensation begins, reverse order:
  1. release_reservation() -- SUCCEEDS.
     Inventory service confirms the reservation
     is released.
  2. refund_charge() -- FAILS. Payments service
     is briefly unreachable (network blip).

Naive handling: the saga's own retry attempt for
step 2 also fails (still unreachable), and with
no further retry configured, the saga just...
stops. The customer was charged, the reservation
was correctly released, but the refund never
happened -- a real, silent inconsistency.

With a reconciliation process: a background job
periodically scans for sagas stuck in a
"compensating" state longer than expected,
finds this one, retries the specific failed
compensating action (refund_charge) with fresh
backoff, and either succeeds or escalates to a
human/alert if it keeps failing.

8: This is the partial-completion scenario itself — one compensating action succeeded, the very next one failed.
14: Without a reconciliation process, this is where the story silently ends — a real inconsistency with no mechanism watching for it.

Why this works: This is the concrete case "partial completion" refers to — a saga can fail even in the middle of cleaning up after its own earlier failure, and treating that as a rare, ignorable edge case is exactly how real production systems accumulate silent, undetected inconsistencies.

Treating compensating actions as guaranteed to succeed on the first try, with no retry policy of their own

Wrong

text

def compensate(saga):
    for step in reversed(saga.completed_steps):
        step.compensating_action()  # called once,
                                      # no retry,
                                      # no idempotency
                                      # check

Better

text

def compensate(saga):
    for step in reversed(saga.completed_steps):
        retry_with_backoff(
            step.compensating_action,
            idempotency_key=f"{saga.id}:{step.name}",
            max_attempts=5,
        )
        # failures past max_attempts flag the saga
        # for reconciliation, not silent abandonment

What you see: The exact same class of failure that made the ORIGINAL saga step need a retry policy (a transient network error, a brief dependency outage) also hits a compensating action, but because compensation was written as a single unretried call, the failure is treated as final instead of transient — leaving a real, permanent inconsistency from what was actually just a temporary blip.

Why: A compensating action is a real network call to a real dependency, exactly like the original step it is undoing — it is subject to the exact same transient-failure modes, and giving it no retry policy while the original step had one is an inconsistent application of the same resilience discipline to two operations that face identical risks.

Compensation, reverse order, itself fails partway

schedule() fails

triggers compensation

release_reservation()

succeeds

refund_charge()

fails — briefly unreachable

Reconciliation job

retries, then alerts

The four operational concerns, and why each is necessary

The four operational concerns, and why each is necessary
Concern	What happens without it	Standard fix
Failure handling	No clear rule for when/how compensation triggers or in what order	Explicit trigger conditions; reverse-order compensation
Idempotency	A duplicate compensation trigger double-refunds or double-releases	Idempotency key / unique constraint on the compensating action itself
Retries	A transient failure in compensation is treated as final, leaving the step un-compensated	Bounded retry with backoff, same as any other unreliable call
Partial completion	A saga stuck mid-compensation goes unnoticed indefinitely	Monitoring, alerting, and a reconciliation process for stuck sagas

Remember: Every saga step that commits a real effect needs an explicit, purpose-built compensating action, run in reverse order on failure. Making this reliable requires the same discipline as any other unreliable operation: idempotency (safe to trigger twice), bounded retries with backoff, and monitoring for sagas stuck mid-compensation — a saga can fail even while cleaning up after its own earlier failure, and that has to be detectable, not silently ignored.