1.Why replicas, queues and distributed services temporarily disagree
Eventual consistency is the property that, given enough time with no new writes, every copy of a piece of data across a distributed system will converge to the same value — but at any given moment before that, different parts of the system can legitimately disagree about the current state. This disagreement is not a bug; it is the direct, structural consequence of propagation taking real time across a network. A replica has not yet applied the primary's latest write (see replication lag), a queue consumer has not yet processed the latest message, and two services that each cache the same entity independently can each be looking at a snapshot from a different moment.
Key features
- Replica lag: a replica applies a primary's writes asynchronously, so it can be milliseconds to seconds behind depending on write volume — a read routed to a lagging replica sees the previous state, not the latest one (the exact mechanism the replication-lag concept covers in depth)
- Queue/message propagation delay: a message published to a queue is not instantly visible everywhere — a consumer has to receive and process it, and different consumers (or consumer groups) can be at different points in that processing at any given moment
- Independent caches: two services that each cache the same underlying entity, refreshed on their own separate schedules or triggers, can each be correctly serving a snapshot from a different point in time
- This disagreement is bounded, not permanent — the defining property of "eventual" consistency is that the system converges once writes stop, distinguishing it from a system that would simply diverge and never agree
- The disagreement window is a real, measurable quantity (often called the "inconsistency window" or "staleness bound"), not an unknowable black box — it can be monitored and, for many systems, bounded with an explicit maximum
Three sources of temporary disagreement, and the mechanism behind each
- Primary → Primary: commits "Alexandra" (t=0.000s)
- Primary → Read replica: still shows "Alex" (t=0.050s)
- Primary → Search index: still shows "Alex" (t=0.200s)
- Primary → Recommendation cache: still shows "Alex" (up to t=5:00)
Signature
disagreement is temporary and structural — propagation takes real time, but the system provably converges
Distributed components disagree temporarily because propagation — replication, queue processing, cache refresh — takes real time, not because anything is broken. "Eventual" means the system provably converges once writes stop; the actual staleness window is a measurable, component-specific quantity worth tracking explicitly, not a vague unknowable delay.