beginnerintermediateadvanced

System Design · Delivery Semantics and Idempotency

Delivery Semantics and Idempotency

Why exactly-once is hard, designing consumers for duplicates, and the five mechanisms that make it real

Three semanticsat-most, at-least, exactly-once
Ack ambiguitylost message vs lost ack look identical
Idempotent consumerssame result no matter the repeat count
Five mechanismskey, constraint, dedup table, state check, outbox

1.At-most-once, at-least-once, and why exactly-once is hard everywhere

This is the general version of the delivery-guarantee trade-off that shows up in every distributed messaging system, not just Kafka: at-most-once may silently lose work, at-least-once may silently duplicate it, and exactly-once — never losing and never duplicating — is fundamentally hard to guarantee once a message crosses any network boundary, because the sender can never be certain the receiver's acknowledgment wasn't lost after the work actually succeeded.

Key features

At-most-once: send/process without confirming — never duplicates, but a lost message is gone with no automatic recovery
At-least-once: retry until confirmed — never silently loses work, but a lost confirmation causes a real duplicate delivery, not just a theoretical one
The reason exactly-once is hard: the sender cannot distinguish "the message was lost" from "the message succeeded but the acknowledgment was lost" — both look identical (no ack received) from the sender's side
True exactly-once delivery would require the sender and receiver to agree atomically on outcome across a network — which for two independent systems generally cannot be done without one of them accepting extra coordination cost (e.g. Kafka's transactions, scoped to Kafka-to-Kafka)
The practical, achievable target for almost every real system is at-least-once delivery combined with idempotent processing — producing the effect of exactly-once without requiring the delivery layer itself to guarantee it

The three semantics, generalized across any distributed messaging system

The three semantics, generalized across any distributed messaging system
Semantic	Can lose?	Can duplicate?	How achieved
At-most-once	Yes	No	Send/process without retry or confirmation
At-least-once	No	Yes	Retry until acknowledged
Exactly-once (effective)	No	No (in effect)	At-least-once delivery + idempotent processing

Indistinguishable from A's side: both are a timeout

Message lost

Request never reaches B (network drop)
B never processes the payment
A sees a timeout — no response

Ack lost

B receives it, processes it, sends 200 OK
The response is lost on the way back
A sees the exact same timeout

Signature

at-most-once: may lose · at-least-once: may duplicate · exactly-once: needs idempotency, not just delivery

At-most-once can silently lose work; at-least-once can silently duplicate it; true exactly-once is fundamentally hard because a sender can never tell "message lost" apart from "acknowledgment lost." The practical target is at-least-once delivery plus idempotent processing.

2.Designing consumers to be idempotent

An idempotent operation produces the same end result no matter how many times it is applied — calling it once or five times with the same input leaves the system in the identical final state. Since at-least-once delivery makes duplicate messages a normal, expected occurrence rather than a rare edge case, a consumer that will receive messages under at-least-once semantics needs to be designed idempotent from the start, not patched to handle duplicates after a production incident reveals the gap.

Key features

Idempotent: applying the operation N times has the same effect as applying it once — this is a property of the operation, not of the message or the broker
A duplicate message under at-least-once delivery is not a rare failure case — it is an expected, normal occurrence that will happen in production
"balance = balance - 10" is NOT idempotent — running it twice subtracts 20. "balance = 90" (an absolute set) IS idempotent — running it twice still leaves balance at 90
Idempotency has to be designed at the level of the actual state change, not assumed from "the message has a unique ID" alone — the ID alone does nothing unless the consumer actively checks it
Idempotent design is a property to build in during the initial implementation — retrofitting it after duplicates have already caused a production incident is strictly harder than designing for it from the start

Naturally idempotent vs naturally non-idempotent operations

Naturally idempotent vs naturally non-idempotent operations
Operation	Idempotent?	Why
SET balance = 90	Yes	Same absolute value every time, regardless of repeat count
balance -= 10	No	Each repeat changes the result further
INSERT with a unique constraint on order_id	Yes (effectively)	A duplicate insert fails harmlessly at the database level
INSERT with no uniqueness constraint	No	Each repeat creates a new, duplicate row
Mark status = "shipped"	Yes	Setting the same status again changes nothing further
Send an email	No (by default)	Each call sends another email — needs an explicit dedup mechanism

Idempotent consumer: guard the deduction with a uniqueness check

checkno rowreturnedrow returned

order_placed message

may be a duplicate

INSERT processed_orders

ON CONFLICT DO NOTHING

Skip

already processed

Deduct inventory

genuinely first time

Command

text

-- non-idempotent: relative change
UPDATE accounts SET balance = balance - 10 WHERE id=1;

-- idempotent: absolute set, or guarded by a
-- unique/already-processed check
UPDATE accounts SET balance = 90 WHERE id=1;

Idempotency is a property of the operation, not the message — a unique message ID does nothing unless the consumer actively checks it, atomically, against what has already been processed. Design for it from the start; duplicates under at-least-once delivery are normal, not rare.

3.Idempotency keys, unique constraints, dedup tables and outbox/inbox

These are the five concrete mechanisms that actually implement idempotent processing in practice. An idempotency key is a client-generated unique identifier for one logical operation, checked before acting. A unique constraint lets the database itself reject a duplicate. A deduplication table explicitly records processed IDs. A state check verifies current status before acting (skip if already done). The transactional outbox/inbox pattern solves the specific problem of keeping a database write and a message publish (or a message receipt and processing) atomic with each other.

Key features

Idempotency key: a unique value the caller generates per logical operation (not per retry) and sends with every attempt — the receiver checks it before processing
Unique constraint: the database rejects a duplicate insert outright — the simplest mechanism when the operation maps naturally to "insert exactly one row per logical event"
Deduplication table: an explicit table of already-processed IDs, checked (often atomically, via the same unique-constraint mechanism) before acting
State check: verify the current status before acting — e.g. only ship an order if its status is not already "shipped," making a repeat request a safe no-op
Transactional outbox: write business data and an "event to publish" record in the SAME local database transaction, then a separate process publishes it — solves the dual-write problem (what if the DB write succeeds but the message publish fails, or vice versa?)
Inbox (the consumer-side mirror of outbox): record each processed message ID in the same transaction as the resulting state change, so a duplicate delivery can be detected atomically with the actual work

Five mechanisms and what each is naturally good for

Five mechanisms and what each is naturally good for
Mechanism	Best fit	Key property
Idempotency key	Client-initiated operations (API requests) that may be retried	Caller controls the key, reused across retries of the same logical operation
Unique constraint	Operations that map to "insert exactly one row"	Database enforces it atomically, no application race condition
Deduplication table	Consumer-side message processing	Explicit, inspectable record of what has been handled
State check	Operations with a natural status field	No extra table needed — the existing state IS the check
Outbox/inbox	Keeping a DB write and a message atomic with each other	Solves the dual-write problem specifically

Five idempotency mechanisms

Idempotency key

client-generated, reused per retry

Unique constraint

DB rejects a duplicate insert

Dedup table

explicit processed-IDs record

State check

skip if already done

Outbox/inbox

DB write + publish, atomically

Command

text

-- idempotency key on an API request
POST /charges
Idempotency-Key: a1b2c3-retry-safe-key
{ "amount": 1000, "customer": "cus_1" }

-- outbox: one local transaction, two tables
BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox_events (...) VALUES (...);
COMMIT;
-- a separate publisher process reads outbox_events
-- and publishes them, marking each as sent

Idempotency key for client-retried operations, unique constraint for "insert exactly once," a dedup table or state check on the consumer side, and the outbox/inbox pattern specifically for keeping a database write and a message atomic with each other.

Spotted a mistake on this page? Tell us.

Read Sheet