beginnerintermediateadvanced

System Design · Section 35

Distributed Transactions

Level: advanced
Read: 22 min
Concepts: 3

A single database's ACID guarantee only covers operations it directly manages — there is no built-in mechanism that makes a write in one database part of the same atomic, isolated unit as a write in a completely independent database, which is the structural reason cross-service transactions are hard, not a solvable engineering gap. Two-phase commit genuinely achieves cross-service atomicity through a prepare/vote phase followed by a commit-or-abort phase, at a real cost: every participant holds its locks for the entire window between the two phases, and a coordinator that crashes after collecting all yes votes but before sending the final decision leaves every participant blocked indefinitely with no safe way to decide unilaterally. Because of that cost and fragility, sagas — a sequence of independently-committing local transactions plus explicit compensating actions to unwind a partial failure — are generally preferred, trading 2PC's locking for visible intermediate states and compensation logic that has to be deliberately designed per step, not assumed to exist automatically.

This section

Two Phase Commit to power Distributed Transactions in a Distributed SystemArpit Bhayani

What is true here

Cross-service ACID is structurally hard — each database is only ACID with itself, with no shared commit boundary spanning two independent databases.
Two-phase commit achieves real cross-service atomicity via prepare/vote then commit/abort, at the cost of every participant holding locks for the whole window.
A coordinator crash after collecting all yes votes but before the final decision is 2PC's critical fragility — participants can be left blocked indefinitely.
Sagas trade 2PC's locking for a sequence of independently-committing local transactions plus explicit, per-step compensating actions — not automatic symmetric undos.

What you will be able to do

Explain why no single database's ACID guarantee extends across service boundaries
Trace two-phase commit through both phases and identify the exact coordinator-failure scenario that makes it fragile
Explain the saga alternative's trade-off — visible intermediate states and designed compensation, in exchange for no cross-service locking

Why the problem exists, and 2PC's answer to it

Why cross-service ACID is structurally hard, and the two-phase commit protocol's real mechanics and cost.

Why cross-service ACID transactions are difficult

coreadvanced

A single database can give ACID guarantees (atomicity, consistency, isolation, durability) across multiple writes within that one database, because it controls the locking, logging and commit process for everything it manages. A workflow spanning multiple independent services, each with its own database, has no single component in that position — no one process can atomically lock rows across two different databases, no one process can guarantee both writes commit or both roll back together, because each database only knows how to be ACID with itself, not in coordination with another database it has never heard of.

Think of it as

A single restaurant kitchen can guarantee that an order either fully comes out together or gets fully cancelled — one head chef coordinates everything happening at every station, so "the burger is ready but the fries are not" simply is not allowed to reach the table. Now imagine an order that needs a dish from a completely separate restaurant across town, with its own kitchen, its own staff, and no shared coordination. There is no single head chef who can force both kitchens to either both finish or both cancel together — the best you can do is have someone run between them relaying messages, and there is always a window where one kitchen has finished and the other has not, with no way to instantly and atomically force agreement across two places that were never designed to coordinate.

What we're doing: Show a two-service order flow where a naive sequential approach leaves the system in a genuinely inconsistent state with no database-level mechanism to prevent it.

cross-service-inconsistency.txttext

Placing an order requires two independent writes,
in two independent services with two independent
databases:
  1. Orders service: INSERT INTO orders (...)
  2. Inventory service: UPDATE inventory
       SET stock = stock - 1 WHERE sku = 'X'

Naive sequential approach (no coordination):
  Step 1 succeeds: the order row is committed in
    the Orders service's own database.
  Step 2 fails: the Inventory service is briefly
    unreachable (network blip, deploy in progress).

Result: an order now exists that was never actually
matched by an inventory decrement -- the two
databases are now genuinely, permanently
inconsistent with each other, and NEITHER
database's own ACID guarantee did anything wrong.
Each one is perfectly consistent WITH ITSELF; the
inconsistency exists only between them, a space
neither database's transaction boundary covers.

9: Nothing about this step is wrong on its own — the Orders database's own ACID guarantee is fully intact.
14: This is the actual inconsistency: it lives BETWEEN two databases, in a space no single database's transaction mechanism was ever designed to cover.

Why this works: This is the precise, structural reason cross-service ACID is hard — the inconsistency does not come from either database failing to be ACID, it comes from there being no larger transaction boundary that spans both of them at all.

Assuming careful sequential ordering of writes is equivalent to atomicity

Wrong

text

"We always write to Orders first, then
Inventory, in that specific order — that keeps
things consistent."

Better

text

"Sequential ordering doesn't make the two
writes atomic — it just determines which
database ends up with the 'orphan' write if the
second one fails. We need an explicit pattern
(two-phase commit, or more commonly a saga with
compensating actions) to actually handle that
failure case, not just a fixed write order."

What you see: A "consistent by convention" cross-service write order works fine in testing and under normal conditions, then produces a real orphaned record the first time the second service is briefly unreachable in production — the specific failure the fixed ordering never actually protected against, only made slightly more predictable in shape.

Why: A fixed write order determines which side fails first when something goes wrong, but does nothing to prevent that failure from happening or to undo the first write once the second one fails — true atomicity requires an explicit mechanism to detect the partial failure and either complete or reverse it, which sequential ordering alone does not provide.

No shared commit boundary between two databases

INSERT order

Orders service, own DB

Order committed

fully consistent, alone

UPDATE inventory

briefly unreachable

What a single database's ACID guarantee covers vs a cross-service operation

What a single database's ACID guarantee covers vs a cross-service operation
Property	Single database	Cross-service operation
Atomicity	Guaranteed — one commit log, one decision	No shared commit boundary across services
Isolation	Row/table locks hide in-progress work	No cross-service lock mechanism exists by default
Failure visibility	A crash is locally detectable	A service can be unreachable, not just down — ambiguous from the outside
Coordination cost	Built into the database engine	Must be explicitly built (2PC, saga, or accepted as a risk)

Remember: A single database's ACID guarantee only covers what it directly manages — there is no built-in mechanism for making a write in one database part of the same atomic, isolated unit as a write in a completely separate database. The inconsistency risk lives in the space between services, which no single database's own transaction boundary was ever designed to cover.

Two-phase commit, and why it is expensive and fragile

coreadvanced

Two-phase commit (2PC) is a protocol that lets a coordinator get all participating services to agree on committing or aborting a transaction together. In the first phase (prepare/vote), the coordinator asks every participant to prepare the transaction and vote yes or no; each participant locks its resources and votes. In the second phase (commit), if every participant voted yes, the coordinator tells everyone to commit; if any participant voted no, the coordinator tells everyone to roll back. It genuinely does provide atomicity across services — the cost is that every participant has to hold its locks and stay blocked between the two phases, and if the coordinator itself fails after collecting votes but before sending the final decision, participants can be left blocked indefinitely, unable to release their locked resources.

Think of it as

It is like a group of friends splitting a large group purchase where every single person has to confirm they have their share of the money ready before anyone actually pays. In phase one, an organizer goes around asking everyone "do you have your share ready?" and each person sets that money aside, unavailable for anything else, while they wait for the group's final answer. In phase two, once everyone has confirmed, the organizer says "go" and everyone pays at once — or if even one person says no, the organizer tells everyone to stand down and return the set-aside money. The real cost: while everyone is waiting between "I've set my share aside" and the final "go" signal, that money is locked up and unusable for anything else, and if the organizer disappears after collecting everyone's confirmations but before giving the final signal, everyone is left standing there indefinitely with their money locked up, unable to decide on their own whether to pay or stand down.

text

Coordinator -> P1, P2, P3: PREPARE
P1, P2, P3  -> Coordinator: VOTE (yes/no)

if all votes == yes:
    Coordinator -> P1, P2, P3: COMMIT
else:
    Coordinator -> P1, P2, P3: ABORT

What we're doing: Show the coordinator-failure scenario that makes 2PC fragile — participants blocked indefinitely after all voting yes.

coordinator-failure.txttext

3 participants (P1, P2, P3), 1 coordinator.

Phase 1:
  Coordinator sends PREPARE to P1, P2, P3.
  All three lock their resources, log the pending
  transaction, and vote YES.
  Coordinator has now collected 3/3 yes votes.

Coordinator CRASHES here, before sending phase 2's
COMMIT decision to anyone.

Phase 2 (never happens):
  P1, P2, P3 are all sitting with resources locked,
  having voted yes, waiting for a COMMIT or ABORT
  message that will never arrive from this
  coordinator.

  None of them can safely decide on their own:
    - Committing without confirmation risks
      committing a transaction the coordinator
      might have actually decided to abort (if it
      saw a NO from a participant this one never
      knew about).
    - Aborting without confirmation risks aborting
      a transaction the coordinator had already
      told at least one OTHER participant to commit.

  Every participant stays blocked, resources locked,
  until the coordinator recovers (if it ever does)
  or a human intervenes.

8: This is the exact moment of vulnerability — every participant has committed to a decision it cannot reverse alone, and the only entity that knows the real outcome is gone.
19: Neither option (commit or abort unilaterally) is actually safe — this is precisely why the protocol calls this state "blocked" rather than something a participant can resolve independently.

Why this works: This is the concrete mechanism behind "2PC is fragile" — it is not a vague warning, it is this specific, well-defined failure window where a coordinator crash at exactly the wrong moment leaves every participant unable to make a safe, unilateral decision.

Assuming a participant can safely guess the coordinator's decision after a timeout instead of staying blocked

Wrong

text

# participant, after voting yes and waiting
# past some timeout with no response:
if timeout_exceeded:
    commit_anyway()  # or abort_anyway() — either
                       # guess can be wrong

Better

text

# correct behavior per the protocol: remain
# blocked, holding locks, until a real decision
# is received from the coordinator (or a
# recovery/consensus mechanism resolves it) —
# there is no safe unilateral guess

What you see: Two participants that both timed out and each guessed independently (one committed, one aborted) leave the overall transaction in exactly the split, inconsistent state 2PC exists to prevent — the timeout-based guess reintroduces the very failure mode the blocking behavior was designed to avoid.

Why: A participant genuinely cannot tell, from a timeout alone, whether the coordinator crashed after deciding to commit or after deciding to abort — guessing either way has a real chance of being wrong, and unlike a normal application bug, a wrong guess here directly produces the cross-service inconsistency the whole protocol exists to prevent.

Two-phase commit: prepare, then a coordinator crash

Coordinator

1. PREPARE
2. PREPARE
3. PREPARE
4. vote yes
5. vote yes
6. vote yes
7. crashes3/3 yes collected, decision never sent — all locked, blocked indefinitely

The two phases of 2PC

The two phases of 2PC
Phase	Coordinator action	Participant action
1: Prepare/vote	Asks every participant to prepare and vote	Locks resources, logs the pending transaction, votes yes/no
2: Commit/abort	Collects votes; tells all to commit (all-yes) or abort (any-no)	Commits and releases locks, or rolls back using its log

Remember: Two-phase commit genuinely achieves cross-service atomicity: prepare/vote, then commit-or-abort based on unanimous agreement. The cost is real — every participant holds its locks for the entire window between the two phases, and a coordinator crash after collecting all yes votes but before sending the final decision leaves every participant blocked indefinitely, unable to safely decide on its own.

The generally preferred alternative

Why sagas, local transactions and compensating actions are usually chosen over 2PC, and the trade-off that choice carries.

Preferring sagas, local transactions and compensating actions over 2PC

standardadvanced

Rather than trying to make a cross-service operation atomic upfront the way two-phase commit does, a saga breaks it into a sequence of local transactions — each one fully committed within its own service, with no cross-service locks held at any point. If a later step fails, the saga does not roll back a single distributed transaction; instead, it runs compensating actions that undo the effect of the earlier steps that already committed. This trades 2PC's strong, immediate cross-service atomicity for a design where each individual step is fast and non-blocking, at the cost of the system passing through real intermediate states that a saga's compensation logic has to be built to handle correctly.

Think of it as

Booking a multi-leg trip through separate airlines with no unified booking system is a saga in the real world. You book the first flight — that booking is fully final and committed the moment you pay, no cross-airline lock is held. Then you book the second flight. If the second airline has no seats left, there is no "distributed rollback" across both airlines' systems — instead, you have to take a compensating action yourself: cancel the first flight (which may cost a fee, may not be instant, and may not perfectly restore your original state, e.g. a lost cancellation fee). Each booking was a genuine, independent local transaction; recovering from a failure partway through means explicitly undoing what already happened, not pretending it never happened at all.

text

# saga: sequence of local transactions + compensations
step 1: charge_card()         compensate: refund_card()
step 2: reserve_inventory()   compensate: release_inventory()
step 3: schedule_shipping()   compensate: cancel_shipping()

# if step 3 fails after 1 and 2 already committed:
run compensate for step 2, then compensate for step 1
(in reverse order of the original steps)

What we're doing: Show a 3-step order saga where the last step fails, and the compensating actions that unwind the already-committed earlier steps.

saga-compensation.txttext

Order flow, as a saga of 3 local transactions:
  1. Payments service: charge the customer's card.
     COMMITS independently -- the charge is real
     and final the moment this step succeeds.
  2. Inventory service: reserve the item.
     COMMITS independently -- the reservation is
     real, another order cannot claim this unit.
  3. Shipping service: schedule a shipment.
     FAILS -- no carrier capacity available today.

Recovery (compensating actions, in reverse order):
  Compensate step 2: release the inventory
    reservation -- the item becomes available to
    other orders again.
  Compensate step 1: refund the customer's card --
    a NEW operation (a refund), not an undo of the
    original charge, which already fully happened
    and cannot be un-happened, only reversed.

End state: customer was briefly charged and then
refunded (a real, observable intermediate state --
their bank statement may show both transactions),
inventory was briefly reserved and then released.
Nothing was ever "rolled back" in the database
sense -- everything that happened, happened for
real, and was then explicitly reversed.

4: This step is genuinely final the moment it commits — there was no cross-service lock holding it back, unlike 2PC's prepare phase.
16: A refund is a new, separate operation — not a rollback — which is the concrete difference between compensation and a database transaction rollback.

Why this works: This is the real trade-off in action — no participant was ever blocked waiting on a coordinator, but the system genuinely passed through a state (charged, then refunded) that a database transaction would never expose, and that state has to be something the design explicitly accounts for.

Assuming a compensating action is a clean, symmetric undo of the original step

Wrong

text

"The compensating action for 'send confirmation
email' is just 'don't send it' — easy, symmetric
undo."

Better

text

"If the email already sent before the saga
needed to compensate, there's no way to un-send
it — the compensating action might instead be
'send a follow-up cancellation email,' which is
NOT symmetric with the original action and needs
to be designed as its own explicit step, not
assumed to exist for free."

What you see: A saga's compensation logic is written as if every step has a perfect, automatic inverse, and the design breaks down the first time a real step (an email already sent, a physical item already shipped, a webhook already delivered to a third party) turns out to have no clean undo — the compensating action has to be actively designed, not derived mechanically from the forward action.

Why: Compensating actions are ordinary application logic that has to be deliberately designed for each specific step, because many real actions (sending a notification, calling an external API that took immediate effect) genuinely have no symmetric inverse — treating compensation as an automatic byproduct of the forward action rather than its own designed step is a common and costly saga-design mistake.

Two-phase commit vs. saga

Two-phase commit

+Locks held for the whole protocol duration
+Atomic from the outside, no visible intermediate states
+Coordinator crash can block every participant indefinitely

Saga

−Each local transaction commits independently
−Real intermediate states are visible while in progress
−A late failure runs explicit compensating actions

2PC vs saga, the same underlying trade-off

2PC vs saga, the same underlying trade-off
Property	Two-phase commit	Saga
Resources locked across services?	Yes, for the whole protocol duration	No — each local transaction commits independently
Intermediate states visible?	No — atomic from the outside	Yes — real intermediate states exist while in progress
Recovery from a late failure	Coordinated rollback (or blocking if the coordinator fails)	Explicit compensating actions, run per completed step
Failure mode under coordinator/participant crash	Can block indefinitely	Degrades to "needs the remaining compensations to run" — recoverable without blocking

Remember: A saga trades 2PC's cross-service locking for a sequence of independently-committing local transactions plus compensating actions to unwind a partial failure. Each step is genuinely final the moment it commits, real intermediate states are visible while the saga is in progress, and compensating actions are new operations designed per step, not automatic symmetric undos.