Filter concepts by levelShowing all levels.

System Design · Eventual Consistency

Eventual Consistency

Why distributed components disagree temporarily, choosing per workflow, and modeling async completion honestly

  • Temporary disagreementpropagation takes real time, by construction
  • Per-workflow choicesame data, different consistency by use
  • Explicit statuspending/processing/complete/failed — not a guess
  • Converges provablythe guarantee "eventual" actually makes

1.Why replicas, queues and distributed services temporarily disagree

Eventual consistency is the property that, given enough time with no new writes, every copy of a piece of data across a distributed system will converge to the same value — but at any given moment before that, different parts of the system can legitimately disagree about the current state. This disagreement is not a bug; it is the direct, structural consequence of propagation taking real time across a network. A replica has not yet applied the primary's latest write (see replication lag), a queue consumer has not yet processed the latest message, and two services that each cache the same entity independently can each be looking at a snapshot from a different moment.

Key features

  • Replica lag: a replica applies a primary's writes asynchronously, so it can be milliseconds to seconds behind depending on write volume — a read routed to a lagging replica sees the previous state, not the latest one (the exact mechanism the replication-lag concept covers in depth)
  • Queue/message propagation delay: a message published to a queue is not instantly visible everywhere — a consumer has to receive and process it, and different consumers (or consumer groups) can be at different points in that processing at any given moment
  • Independent caches: two services that each cache the same underlying entity, refreshed on their own separate schedules or triggers, can each be correctly serving a snapshot from a different point in time
  • This disagreement is bounded, not permanent — the defining property of "eventual" consistency is that the system converges once writes stop, distinguishing it from a system that would simply diverge and never agree
  • The disagreement window is a real, measurable quantity (often called the "inconsistency window" or "staleness bound"), not an unknowable black box — it can be monitored and, for many systems, bounded with an explicit maximum

Three sources of temporary disagreement, and the mechanism behind each

Three sources of temporary disagreement, and the mechanism behind each
SourceMechanismTypical staleness window
Replica lagAsynchronous replication applies writes with a delayMilliseconds under normal load; seconds+ under write bursts
Queue/message delayA consumer has not yet received or processed a published messageDepends on consumer throughput and backlog depth
Independent cachesEach cache refreshes on its own schedule, unaware of the othersBounded by each cache's own TTL or invalidation trigger
One write, three different convergence times
Primary
Read replica
Search index
Recommendation cache
  1. 1commits "Alexandra"t=0.000s
  2. 2still shows "Alex"t=0.050s
  3. 3still shows "Alex"t=0.200s
  4. 4still shows "Alex"up to t=5:00
  1. Primary → Primary: commits "Alexandra" (t=0.000s)
  2. Primary → Read replica: still shows "Alex" (t=0.050s)
  3. Primary → Search index: still shows "Alex" (t=0.200s)
  4. Primary → Recommendation cache: still shows "Alex" (up to t=5:00)

Signature

disagreement is temporary and structural — propagation takes real time, but the system provably converges

Distributed components disagree temporarily because propagation — replication, queue processing, cache refresh — takes real time, not because anything is broken. "Eventual" means the system provably converges once writes stop; the actual staleness window is a measurable, component-specific quantity worth tracking explicitly, not a vague unknowable delay.

2.Choosing eventual vs strong consistency per workflow

Consistency is not a single system-wide setting to pick once — different workflows within the same system tolerate staleness very differently, and the right choice is made per workflow, not globally. A social media "like count" being briefly stale is invisible to users and costs nothing real. An account balance being briefly stale can mean a user double-spends money that was never really there. The test is: what is the actual, concrete cost if this specific read is a few seconds (or minutes) out of date — and does that cost matter enough to pay strong consistency's real price (more coordination, more latency, less availability during a partition) for this specific piece of data.

Key features

  • Eventually-consistent-acceptable workflows share a pattern: the cost of acting on stale data is small, temporary, and usually self-correcting on the next read — like counts, view counts, "last seen" timestamps, search index freshness, recommendation feeds
  • Strong-consistency-required workflows share the opposite pattern: acting on stale data can cause an irreversible, costly, or safety-critical mistake — account balances, inventory reservations that prevent overselling, anything enforcing a uniqueness or limit invariant (one account per email, no double-booking a seat)
  • The same underlying data can need different consistency for different reads of it — an account balance shown on a dashboard (eventual is fine, it is informational) versus the same balance checked at the moment of an actual withdrawal (must be strongly consistent, it is enforcing a real constraint)
  • Strong consistency has a real, unavoidable cost: more coordination between nodes, generally higher latency, and reduced availability during a network partition (the system may have to refuse an operation rather than risk an inconsistent result) — this cost has to be worth paying for the specific workflow, not assumed free
  • A useful test: ask "what is the worst realistic consequence if this specific read is 5 seconds stale?" — if the honest answer is "nothing a user would notice or that costs the business anything," eventual consistency is very likely fine; if the answer involves money, safety, or a broken invariant, it very likely is not

Matching workflows to the right consistency level

Matching workflows to the right consistency level
WorkflowConsistency needWhy
Like/view counter displayEventualBeing briefly off by a few counts costs nothing and self-corrects
Search result freshnessEventualA newly-created item missing from search for a few seconds is a minor UX gap, not a correctness bug
Account balance display (informational)EventualShown for awareness; the actual constraint is enforced at withdrawal time, not display time
Withdrawal / balance check at transaction timeStrongActing on a stale balance can let a withdrawal succeed that should have been rejected
Inventory decrement at checkoutStrongA stale stock count can oversell a limited item
Unique username/email registrationStrongTwo concurrent signups both seeing "available" can both succeed, violating uniqueness
Same balance, two different consistency needs

Dashboard display

  • Reads from a replica, may be stale
  • Worst case: a slightly outdated number
  • Eventual consistency is fine

Withdrawal request

  • Must read the current, authoritative balance
  • Worst case: a real, unrecoverable overdraft
  • Strong consistency is required
  • Dashboard display
    • Reads from a replica, may be stale
    • Worst case: a slightly outdated number
    • Eventual consistency is fine
  • Withdrawal request
    • Must read the current, authoritative balance
    • Worst case: a real, unrecoverable overdraft
    • Strong consistency is required

Signature

ask: what is the real cost if THIS read is 5 seconds stale? — decide per workflow, not system-wide

Consistency is chosen per workflow, not system-wide — the same data can be read with eventual consistency for informational display and strong consistency for the operation that actually enforces a constraint. Ask what the real cost of staleness is for THIS specific read before defaulting either way.

3.Using explicit states and user-visible status for async completion

When a workflow is genuinely asynchronous — its final result is not available the instant the user takes an action — the right fix is not to hide that fact, but to model it explicitly with a status the user (or calling system) can actually see and act on. Instead of pretending an operation is instantly "done," an explicit state machine (e.g. pending → processing → complete, or pending → failed) tells the truth about where the operation currently stands, and gives the UI or caller something concrete to poll, subscribe to, or display, rather than silently showing stale or misleading information as if it were final.

Key features

  • An explicit state field (e.g. status: "pending" | "processing" | "complete" | "failed") turns an invisible asynchronous gap into a visible, queryable fact instead of an implicit, undocumented assumption
  • Without an explicit status, a caller has no way to distinguish "still processing, be patient" from "silently failed, will never complete" from "actually already done, but this specific read is stale" — three very different situations that look identical without an explicit signal
  • A well-designed state machine names every real outcome, not just the happy path — pending, processing, complete AND failed/cancelled/expired, so a caller can build correct handling for every state rather than assuming "not complete yet" always means "still in progress"
  • Explicit status pairs naturally with polling or a push mechanism (webhook, WebSocket, SSE) so a caller does not have to guess when to check again — the status itself, or a change notification, drives the next action
  • This pattern directly prevents a common asynchronous-workflow bug: a caller that assumes "I submitted this, so it must be done by the time I check again a moment later" and acts on an assumption instead of an actual observed state

Implicit assumption vs explicit status for the same async workflow

Implicit assumption vs explicit status for the same async workflow
PropertyNo explicit statusExplicit status field
Caller can tell "still working" from "failed"No — both look like "not done yet"Yes — distinct states
UI can show meaningful progressNo — has to guess or show nothingYes — renders the actual current state
A caller polling too early getsAn ambiguous or misleading resultA clear "pending" or "processing" answer
Debugging a stuck workflowHard — no record of where it got stuckEasy — the last recorded state shows exactly where
Video processing: explicit states, not a timing guess
transcodingstartssucceedsfails

Uploaded

start

Processing

Ready

end

Failed

end

  • Uploaded (start)
    • → Processing when transcoding starts
  • Processing
    • → Ready when succeeds
    • → Failed when fails
  • Ready (end)
  • Failed (end)

Command

json
{
  "orderId": "ord_123",
  "status": "processing",
  "statusHistory": [
    { "status": "pending", "at": "2026-08-22T10:00:00Z" },
    { "status": "processing", "at": "2026-08-22T10:00:02Z" }
  ]
}

When completion is genuinely asynchronous, model it with an explicit, user-visible status (including real failure/cancellation states, not just the happy path) instead of hiding the delay or letting a caller infer status from elapsed time — the status itself is the honest signal, timing alone is not.

Remember

  • Disagreement between replicas, queues and caches is structural, not a bug — propagation takes real time.
  • "Eventually" is a measurable, component-specific staleness window, not a vague unknowable delay.
  • Consistency is chosen per workflow — the same data can need eventual for display and strong for the operation enforcing a real constraint.
  • Model genuinely asynchronous completion with an explicit status (including real failure states), never an inferred timing guess.

Pro tip

  • Ask "what is the real cost if this specific read is 5 seconds stale?" before defaulting to either consistency level.
  • Route only the workflows that genuinely need strong consistency through it — applying it everywhere pays a real cost for reads that never needed the guarantee.

Next up

  • Distributed transactions
  • Consistency models
  • CAP theorem

Sources

  • System Design Primer — github.com/donnemartin/system-design-primer
  • Checked 2026-08-22

Spotted a mistake on this page? Tell us.