Event Ordering and Duplicate Handling

Global ordering means every event in a system, across every partition, every producer, and every machine, can be placed in one single, agreed-upon sequence — event 1 happened before event 2, which happened before event 3, no exceptions, no ambiguity, regardless of which server produced which event. Guaranteeing this is expensive because it requires coordination: every producer has to agree, in real time, on its position in one global sequence, which typically means funneling all writes through a single point (eliminating the parallelism a distributed system exists to provide) or running an expensive consensus protocol on every single event. This is why most real distributed messaging systems (Kafka included) deliberately do not offer global ordering as a default guarantee — instead, they guarantee ordering only within a partition (or equivalently, within all events sharing the same key), which is dramatically cheaper because a single partition can be owned and sequenced by one machine independently of every other partition, with no cross-partition coordination needed at all. The trade-off is real: two events for the same key are strictly ordered, but two events for different keys have no defined relative order, even if one was produced well before the other in wall-clock time. Most real workloads do not actually need global ordering — they need ordering per-entity (all changes to order #42 must apply in the order they happened; the relative order of a change to order #42 versus order #99 usually does not matter) — and partition-level ordering delivers exactly that, at a small fraction of the cost.

# Partitioning by entity key: order-level ordering
# without any global coordination
producer.send(topic="order-events", key=order_id,
               value=event)
# All events for order_id=42 land in the same
# partition and are strictly ordered relative to
# each other. Events for order_id=99 may land in a
# different partition and have no defined ordering
# relative to order_id=42's events -- which is fine,
# because nothing in the application logic ever
# needs to compare the relative order of two
# different orders.

At-least-once delivery — the standard guarantee most real messaging systems provide, because it is far cheaper than exactly-once — means a consumer will occasionally see the same event more than once (a producer retries after a network blip that actually succeeded, a consumer crashes after processing but before acknowledging), and a consumer that is not written to expect this will apply an event's effect twice. Even within a single partition's strict ordering guarantee, a consumer can still encounter events effectively out of order relative to its own processing — a rebalance, a retry, or a slow consumer that falls behind and gets events replayed from an earlier offset can all present events to application logic in a sequence that does not match production order, even when the underlying log itself preserved it correctly. Because both of these are normal, expected behavior of a real distributed system rather than rare edge cases, consumer logic has to be written defensively from the start: idempotent processing (applying the same event twice produces the same result as applying it once) handles duplicates, and order-tolerant processing (using a version number, timestamp, or explicit ordering key rather than assuming "the order I received it in is the order it happened") handles the out-of-order case. A consumer that instead assumes "each event arrives exactly once, in the exact order it happened" is not describing a rare failure mode when that assumption breaks — it is describing the everyday operating condition of the delivery guarantee it is actually built on.

# A consumer defended against both duplicates and
# out-of-order arrival
def handle_event(event):
    if event.id in already_processed:      # duplicate defense
        return
    current = get_current_state(event.entity_id)
    if current and current.version >= event.version:
        already_processed.add(event.id)     # older/duplicate:
        return                              # skip, don't overwrite
    apply(event)
    already_processed.add(event.id)

Once a consumer accepts that arrival order is not a reliable signal of true ordering (the prior concept), it needs an actual mechanism to determine which of two events is genuinely newer, and three mechanisms cover almost every case. An event timestamp records when an event happened, typically set by the producer at creation time, and lets a consumer compare "which of these two events happened first" directly — but timestamps from different machines are only as reliable as those machines' clocks, and clock drift (covered in the next section) can make two genuinely-ordered events appear to have times in the wrong order. A sequence number is a monotonically increasing counter, usually scoped to a specific entity or partition, that a producer increments on every event for that entity — unlike a timestamp, it has no dependency on wall-clock accuracy at all, since it is just a counter that only ever goes up, making it a stronger ordering signal wherever it is available. A version check (often called optimistic concurrency control) is the version-number variant already used for idempotency in the prior concept, applied specifically as an ordering gate: before applying an update, a consumer compares the incoming event's version against the current stored version, and only applies it if the incoming version is actually newer, which is what makes stale, late-arriving events safe to simply discard rather than needing to be correctly sequenced in the first place. In practice, sequence numbers or version checks are usually preferred over raw timestamps wherever an entity has one clear owner producing its events, precisely because they sidestep the clock-reliability problem timestamps carry.

# Three signals recorded on the same event, each
# serving a different purpose
event = {
    "entity_id": "order-42",
    "sequence": 7,                    # monotonic,
                                       # per-entity
    "version": 7,                     # same value
                                       # here, used
                                       # as an
                                       # optimistic-
                                       # concurrency
                                       # gate
    "produced_at": "2026-08-26T10:03:21.442Z",  # for
                                       # humans/audit
                                       # trails only
}

def apply_update(event, current_state):
    if event["version"] <= current_state.version:
        return  # stale or duplicate -- discard
    current_state.apply(event)
    current_state.version = event["version"]