beginnerintermediateadvanced

System Design · Section 79

Event Ordering and Duplicate Handling

Level: intermediate
Read: 14 min
Concepts: 3

Global ordering — one agreed sequence across every event in a system — requires expensive coordination, either funneling every write through a single point or running consensus on every event, which is why most real messaging systems, Kafka included, guarantee ordering only within a partition or key by default: strict order for events sharing the same key, no defined relative order across different keys. This matches what most real workloads actually need — per-entity ordering, not a global one — as long as the partition key is chosen to match the entity that actually needs ordering rather than chosen only for even load distribution. Because the standard, far cheaper at-least-once delivery guarantee trades "never lose a message" for "occasionally deliver the same message twice," and because rebalances, retries and replays can present even a strictly-ordered partition's events to a consumer out of true sequence, a consumer has to be built from the start to tolerate both conditions as normal operation rather than rare edge cases: idempotent processing (deduplicating by a unique event ID, never by content) absorbs duplicates, and order-tolerant processing (comparing an explicit signal rather than trusting arrival order) absorbs out-of-order arrival. Three mechanisms provide that explicit ordering signal: event timestamps compare "which happened first" directly but are only as reliable as producer clocks, which can drift; sequence numbers are a clock-independent monotonic counter, usually scoped per entity, that sidesteps the clock problem entirely; and version checks compare an incoming event's version against the currently stored version and discard anything not strictly newer, without ever needing to reconstruct a true global order at all — which is why sequence numbers and version checks are usually preferred over raw timestamps wherever an entity has one clear producer.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Global ordering requires expensive coordination; partition-level (per-key) ordering is what most systems guarantee by default and what most real per-entity ordering needs actually require.
Choose a partition/sequence key matched to the entity that needs ordering, not a field chosen only for even load distribution.
At-least-once delivery makes duplicate delivery a normal, expected condition — deduplicate by a unique event ID, never by content.
Even a strictly-ordered partition can present events to a consumer out of true sequence due to rebalances, retries or replays — order-tolerant processing is needed in addition to idempotency, not instead of it.
Sequence numbers and version checks are clock-independent and usually preferred over raw timestamps for resolving true event order or discarding stale updates.

What you will be able to do

Explain why global ordering is expensive and why partition-level ordering satisfies most real per-entity ordering needs
Choose a partition or sequence key that matches the entity actually requiring ordering
Design a consumer that safely handles both duplicate and out-of-order event delivery as normal conditions
Choose between an event timestamp, a sequence number, and a version check for a given ordering or conflict-resolution need

Ordering cost and scope

Why global ordering is expensive, and why partition/key-level ordering is what most systems actually guarantee and need.

Why global ordering is expensive

coreintermediate

Global ordering means every event in a system, across every partition, every producer, and every machine, can be placed in one single, agreed-upon sequence — event 1 happened before event 2, which happened before event 3, no exceptions, no ambiguity, regardless of which server produced which event. Guaranteeing this is expensive because it requires coordination: every producer has to agree, in real time, on its position in one global sequence, which typically means funneling all writes through a single point (eliminating the parallelism a distributed system exists to provide) or running an expensive consensus protocol on every single event. This is why most real distributed messaging systems (Kafka included) deliberately do not offer global ordering as a default guarantee — instead, they guarantee ordering only within a partition (or equivalently, within all events sharing the same key), which is dramatically cheaper because a single partition can be owned and sequenced by one machine independently of every other partition, with no cross-partition coordination needed at all. The trade-off is real: two events for the same key are strictly ordered, but two events for different keys have no defined relative order, even if one was produced well before the other in wall-clock time. Most real workloads do not actually need global ordering — they need ordering per-entity (all changes to order #42 must apply in the order they happened; the relative order of a change to order #42 versus order #99 usually does not matter) — and partition-level ordering delivers exactly that, at a small fraction of the cost.

Think of it as

A large restaurant kitchen with twelve separate stations (grill, salad, dessert, ...), each producing its own dishes in the order tickets arrive at that station — the grill station's dishes come out strictly in the order the grill tickets were received, and the salad station's dishes come out strictly in the order salad tickets were received, but there is no rule at all about whether a specific grill dish comes out before or after a specific salad dish ordered around the same time. Enforcing a single global sequence — every dish in the entire kitchen numbered and produced in exactly that overall order, across all twelve stations — would require every station to constantly check in with a central coordinator before plating anything, destroying the entire point of having twelve stations working in parallel. The kitchen works because almost no one actually cares whether their salad came out before or after another table's dessert; they care that their own dish, from start to plating, happened in the right internal sequence.

text

# Partitioning by entity key: order-level ordering
# without any global coordination
producer.send(topic="order-events", key=order_id,
               value=event)
# All events for order_id=42 land in the same
# partition and are strictly ordered relative to
# each other. Events for order_id=99 may land in a
# different partition and have no defined ordering
# relative to order_id=42's events -- which is fine,
# because nothing in the application logic ever
# needs to compare the relative order of two
# different orders.

What we're doing: Trace order #42's and order #99's events through a partitioned topic and confirm which orderings are actually guaranteed.

partition-ordering-trace.txttext

Topic "order-events", 4 partitions, keyed by order_id

order_id=42 events (all hash to partition 1):
  1. OrderCreated   (t=100ms)
  2. PaymentApplied  (t=150ms)
  3. OrderShipped    (t=400ms)
  -> guaranteed to be consumed in exactly this order

order_id=99 events (all hash to partition 3):
  1. OrderCreated    (t=120ms)
  2. OrderCancelled  (t=130ms)
  -> guaranteed to be consumed in exactly this order

Relative order between order 42's PaymentApplied
(t=150ms) and order 99's OrderCancelled (t=130ms)?
  -> NOT guaranteed, even though 99's event happened
     first in wall-clock time -- they are in
     different partitions

5: This is the actual guarantee a consumer can rely on: every event for the same order ID arrives in the exact sequence it was produced, because the partition key ties them to the same partition and Kafka orders within a partition strictly.
14: This is the guarantee that does NOT exist, and it is fine that it does not — no part of a typical order-processing application ever needs to know whether order 42's payment happened before or after order 99's cancellation, because the two orders are independent entities.

Why this works: The design only works because the partition key (order_id) was chosen to match the actual unit the application needs ordering for — if events had instead been partitioned by, say, a round-robin scheme unrelated to order_id, two events for the same order could land in different partitions and lose the exact ordering guarantee the application actually depends on.

Partitioning by a field unrelated to the entity that needs ordering

Wrong

python

# Partitioned by event timestamp bucket, "for even
# distribution" -- but order_id's events can now
# land in different partitions depending on when
# each one happened to be produced
producer.send(topic="order-events",
               key=str(int(time.time() // 60)),
               value=event)

Better

python

# Partitioned by the entity that actually needs
# ordering
producer.send(topic="order-events",
               key=str(order_id),
               value=event)

What you see: A consumer processing OrderShipped before OrderCreated for the same order, because the two events happened to fall into different one-minute timestamp buckets and were routed to different partitions with no ordering relationship between them.

Why: Partitioning "for even distribution" by an arbitrary field optimizes for a real but secondary concern (spreading load evenly across partitions) while silently discarding the one ordering guarantee the application actually needed — a partition key has to be chosen for the entity whose events must stay in order, with load distribution as a secondary constraint satisfied by having enough distinct entity keys, not the primary one.

Global ordering vs. partition-level ordering, for the same event stream

Global ordering

+Every event across every partition placed in one sequence
+Requires a single coordination point or a consensus protocol per event
+Throughput bounded by that coordination cost
+Rarely what the application actually needs

Partition-level ordering

−Events for the same key strictly ordered
−No coordination needed across partitions
−Throughput scales with partition count
−Matches the common real need: per-entity ordering

Global ordering vs. partition-level ordering

Global ordering vs. partition-level ordering
Aspect	Global ordering	Partition-level ordering
Coordination needed	Every producer/partition, on every event	None across partitions — each partition sequences independently
Throughput	Limited by the single coordination point	Scales with the number of partitions
What is guaranteed	One total order across the entire system	Strict order within the same key; no guarantee across keys
Typical default in real systems	Rare, and usually opt-in at real cost	Kafka, most log-based systems, by default

Remember: Global ordering (one sequence across the entire system) requires expensive coordination and is rarely what a real workload needs. Partition-level (or key-level) ordering — strict order within the same key, no guarantee across different keys — is what most systems, including Kafka, actually guarantee by default, and it is enough for the common real requirement: ordering per entity. Choose the partition key to match the entity that actually needs ordering, not for load distribution alone.

Building resilient consumers

Tolerating duplicates and out-of-order events as normal conditions, and the three signals that establish true order.

Consumers must tolerate duplicates and out-of-order events

coreintermediate

At-least-once delivery — the standard guarantee most real messaging systems provide, because it is far cheaper than exactly-once — means a consumer will occasionally see the same event more than once (a producer retries after a network blip that actually succeeded, a consumer crashes after processing but before acknowledging), and a consumer that is not written to expect this will apply an event's effect twice. Even within a single partition's strict ordering guarantee, a consumer can still encounter events effectively out of order relative to its own processing — a rebalance, a retry, or a slow consumer that falls behind and gets events replayed from an earlier offset can all present events to application logic in a sequence that does not match production order, even when the underlying log itself preserved it correctly. Because both of these are normal, expected behavior of a real distributed system rather than rare edge cases, consumer logic has to be written defensively from the start: idempotent processing (applying the same event twice produces the same result as applying it once) handles duplicates, and order-tolerant processing (using a version number, timestamp, or explicit ordering key rather than assuming "the order I received it in is the order it happened") handles the out-of-order case. A consumer that instead assumes "each event arrives exactly once, in the exact order it happened" is not describing a rare failure mode when that assumption breaks — it is describing the everyday operating condition of the delivery guarantee it is actually built on.

Think of it as

A mail carrier delivering registered letters under a policy of "if I am not sure a letter was received, deliver it again rather than risk it being lost" — this guarantees no letter is ever silently lost, at the cost of the recipient sometimes getting the same letter twice. A well-run mailroom does not treat a duplicate letter as a crisis; it checks the letter's own reference number against what it has already filed, recognizes the duplicate, and simply does not re-file it a second time — the duplicate is absorbed cheaply because the mailroom was built expecting it to happen sometimes. A mailroom that instead assumes "I will only ever receive each letter exactly once" has no such check, and it re-files (double-processes) any duplicate it receives, treating the carrier's occasional redelivery — a known, designed-for part of the guarantee — as if it were a bizarre anomaly.

python

# A consumer defended against both duplicates and
# out-of-order arrival
def handle_event(event):
    if event.id in already_processed:      # duplicate defense
        return
    current = get_current_state(event.entity_id)
    if current and current.version >= event.version:
        already_processed.add(event.id)     # older/duplicate:
        return                              # skip, don't overwrite
    apply(event)
    already_processed.add(event.id)

What we're doing: Trace what happens to account balance updates arriving with a duplicate and one out-of-order event, with and without defenses.

balance-update-trace.txttext

Events for account #7, produced in this true order:
  1. SetBalance(version=1, amount=100)
  2. SetBalance(version=2, amount=150)
  3. SetBalance(version=3, amount=120)

Actually delivered to the consumer, in this order
(one duplicate, one late arrival):
  a. SetBalance(version=1, amount=100)
  b. SetBalance(version=3, amount=120)
  c. SetBalance(version=2, amount=150)   <- late/out
     of order
  d. SetBalance(version=3, amount=120)   <- duplicate
     of (b)

11: Without a version check, this late-arriving version=2 event would be applied after version=3 already set the balance to 120, silently overwriting a newer, correct balance with an older, stale one.
13: Without a duplicate check, this event would re-apply the same SetBalance a second time — harmless for a SetBalance specifically (setting to the same value twice is naturally idempotent), but the identical event ID appearing twice is exactly the shape a genuinely damaging duplicate (e.g. "add $50" applied twice) would also take.

Why this works: A consumer with a version check correctly ends the sequence at balance=120 (the true final state) regardless of the actual delivery order, while a consumer with no version check ends at balance=150 (from the late-arriving, stale version=2 event applied last) — a silently wrong final state produced entirely by trusting arrival order over an explicit version.

Applying account balance changes as increments, with no idempotency check

Wrong

python

def handle_deposit(event):
    account = get_account(event.account_id)
    account.balance += event.amount   # applied again
    save(account)                     # on any retry

Better

python

def handle_deposit(event):
    if event.id in processed_deposit_ids:
        return  # already applied, skip
    account = get_account(event.account_id)
    account.balance += event.amount
    processed_deposit_ids.add(event.id)
    save(account)

What you see: A customer's account is credited twice for the same $50 deposit because the payment processor retried a webhook after a network timeout on the first, actually-successful delivery, and the consumer had no way to recognize the retry as the same event rather than a new deposit.

Why: An increment-style update (`balance += amount`) is inherently non-idempotent — applying it twice produces a different result than applying it once, unlike a SetBalance-style update, which happens to tolerate duplicates for free — any handler using increment-style updates needs an explicit idempotency check, because the operation itself provides none.

A duplicate and an out-of-order event, both handled safely

Producer

Queue

Consumer

State Store

1. publish UpdateV3
2. retry: publish UpdateV3 againambiguous failure — duplicate
3. deliver UpdateV1
4. deliver UpdateV3applied — version 3 > current
5. deliver UpdateV3 (duplicate)skipped — already processed
6. write only for genuinely new, in-order updates

Two failure modes and the defense each one needs

Two failure modes and the defense each one needs
Failure mode	Why it happens	Defense
Duplicate delivery	Producer retries after an ambiguous failure; consumer crashes after processing but before acknowledging	Idempotent processing — track processed event IDs, skip repeats
Out-of-order arrival	Consumer rebalance, retry, or replay from an earlier offset	Order-tolerant processing — compare a version/timestamp, not arrival order

Remember: At-least-once delivery guarantees no message is lost at the cost of occasional duplicates, and even ordered partitions can present events out of processing order due to rebalances or replays — both are normal, expected conditions, not rare edge cases. Idempotent processing (dedupe by a unique event ID, not by content) handles duplicates; order-tolerant processing (compare an explicit version or timestamp, not arrival order) handles out-of-order arrival. A consumer needs both defenses together, since an event can be affected by either or both at once.

Timestamps, sequence numbers and version checks

coreintermediate

Once a consumer accepts that arrival order is not a reliable signal of true ordering (the prior concept), it needs an actual mechanism to determine which of two events is genuinely newer, and three mechanisms cover almost every case. An event timestamp records when an event happened, typically set by the producer at creation time, and lets a consumer compare "which of these two events happened first" directly — but timestamps from different machines are only as reliable as those machines' clocks, and clock drift (covered in the next section) can make two genuinely-ordered events appear to have times in the wrong order. A sequence number is a monotonically increasing counter, usually scoped to a specific entity or partition, that a producer increments on every event for that entity — unlike a timestamp, it has no dependency on wall-clock accuracy at all, since it is just a counter that only ever goes up, making it a stronger ordering signal wherever it is available. A version check (often called optimistic concurrency control) is the version-number variant already used for idempotency in the prior concept, applied specifically as an ordering gate: before applying an update, a consumer compares the incoming event's version against the current stored version, and only applies it if the incoming version is actually newer, which is what makes stale, late-arriving events safe to simply discard rather than needing to be correctly sequenced in the first place. In practice, sequence numbers or version checks are usually preferred over raw timestamps wherever an entity has one clear owner producing its events, precisely because they sidestep the clock-reliability problem timestamps carry.

Think of it as

Imagine trying to determine the true order two letters were written in, using three different pieces of evidence: the date handwritten at the top of each letter (a timestamp — reliable only if both writers' calendars/clocks were correct and agreed with each other), a number stamped by the same office's numbering machine on every letter it sends out, always one higher than the last (a sequence number — reliable regardless of any clock, because it is just a counter), or a note on each letter saying "this supersedes any letter you received with a smaller number" (a version check — the receiving office does not even need to reconstruct the true order, it just needs to know whether to accept or discard letter B once letter C has already been accepted). The sequence number and version check both sidestep the fundamental problem with the handwritten date: they never depended on anyone's clock being right in the first place.

python

# Three signals recorded on the same event, each
# serving a different purpose
event = {
    "entity_id": "order-42",
    "sequence": 7,                    # monotonic,
                                       # per-entity
    "version": 7,                     # same value
                                       # here, used
                                       # as an
                                       # optimistic-
                                       # concurrency
                                       # gate
    "produced_at": "2026-08-26T10:03:21.442Z",  # for
                                       # humans/audit
                                       # trails only
}

def apply_update(event, current_state):
    if event["version"] <= current_state.version:
        return  # stale or duplicate -- discard
    current_state.apply(event)
    current_state.version = event["version"]

What we're doing: Compare relying on timestamps versus version checks when two producers' clocks disagree.

clock-disagreement-trace.txttext

Order #42, updated by two different services with
slightly unsynchronized clocks:

Service A (clock running 3s fast):
  UpdateStatus(status="shipped", ts=10:00:13,
               version=5)

Service B (clock accurate):
  UpdateStatus(status="cancelled", ts=10:00:11,
               version=4)

True order of events (by version, which reflects
what actually happened): version=4 (cancelled)
occurred BEFORE version=5 (shipped)

Ordering by timestamp alone: ts=10:00:11
(cancelled) appears before ts=10:00:13 (shipped)
-- happens to agree here, but only by luck; a
larger clock skew would have reversed it

11: This is the actual, reliable signal: version=4 happened before version=5 because the version is a counter incremented by whichever service is authoritative for this order's state, with no dependency on any clock.
15: The timestamp-based ordering happens to agree with the version-based ordering in this specific example, but only because the 3-second clock skew was not large enough to flip it — a slightly larger skew, or a network delay in when service A's event actually arrives, could easily have made the timestamps appear in the wrong order despite the true order being fixed by version.

Why this works: A consumer that trusted timestamps here got lucky; a consumer that used version numbers was correct by construction, regardless of clock skew — the difference matters exactly in the cases where clocks disagree enough to actually flip the apparent order, which a design cannot predict or bound in advance.

Resolving conflicting updates by comparing producer timestamps

Wrong

python

def apply_update(event, current_state):
    if event["produced_at"] > current_state.updated_at:
        current_state.apply(event)  # trusts producer
        # clocks to be accurate and synchronized

Better

python

def apply_update(event, current_state):
    if event["version"] > current_state.version:
        current_state.apply(event)  # no clock
        # dependency at all

What you see: A cancelled order gets silently reverted back to "shipped" because the shipping service's clock was running a few seconds fast, making its shipped-status event appear, by timestamp, to have happened after the cancellation, when in reality the cancellation happened later and should have won.

Why: Comparing timestamps for conflict resolution implicitly assumes every producer's clock is accurate and synchronized with every other producer's clock to a precision finer than the gap between the two events being compared — an assumption that is often false by exactly the margin that flips the outcome, whereas a version number never depended on that assumption to begin with.

Three signals, three different jobs

Event timestamp

Wall-clock time

human-readable, clock-dependent

Sequence number

Monotonic counter

no clock dependency at all

Version check

Compare before applying

discards stale updates without needing global order

Three ordering mechanisms and what determines their reliability

Three ordering mechanisms and what determines their reliability
Mechanism	Reliability depends on	Typical use
Event timestamp	Producer clock accuracy and synchronization	Human-readable ordering, audit trails, approximate ordering across sources
Sequence number	A single producer's ability to increment a counter correctly	Strict per-entity or per-partition ordering, independent of any clock
Version check	The same counter/version, used as a comparison gate before applying an update	Discarding stale/late-arriving updates without needing global order

Remember: Timestamps let you compare "which happened first" but are only as reliable as producer clocks; sequence numbers are a clock-independent monotonic counter, usually per-entity, which sidesteps that problem entirely; a version check compares an incoming event's version against the current stored version and discards anything not strictly newer, without needing to reconstruct global order at all. Prefer sequence numbers or version checks over raw timestamps wherever an entity has one clear owner producing its events, and keep any sequence counter scoped per-entity, not global.