beginnerintermediateadvanced

System Design · Section 30

Delivery Semantics and Idempotency

Level: intermediate
Read: 20 min
Concepts: 3

At-most-once delivery may silently lose work; at-least-once may silently duplicate it; true exactly-once is fundamentally hard across any network boundary because a sender can never distinguish a lost message from a lost acknowledgment of a message that actually succeeded. The practical, achievable target for almost every real system is at-least-once delivery combined with idempotent consumer design — an operation that produces the same end result no matter how many times it runs. Idempotency is implemented concretely through idempotency keys, unique constraints, deduplication tables, state checks, or the transactional outbox/inbox pattern for keeping a database write and a message publish atomic with each other.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

At-most-once may lose work; at-least-once may duplicate it; exactly-once is hard because lost-message and lost-ack look identical to the sender.
The practical target is at-least-once delivery plus idempotent processing, not relying on a delivery mechanism alone.
Idempotency is a property of the operation — a unique ID does nothing unless the consumer atomically checks it before acting.
Idempotency keys, unique constraints, dedup tables, state checks and the transactional outbox/inbox pattern are the concrete implementation mechanisms.

What you will be able to do

Explain why true exactly-once delivery is fundamentally hard across a network boundary
Design a consumer to handle duplicate delivery safely under at-least-once semantics
Choose the right concrete idempotency mechanism for a given operation shape
Recognize and avoid the dual-write problem between a database write and a message publish

The fundamental trade-off

Why exactly-once delivery is hard everywhere, and what a consumer designed for that reality looks like.

At-most-once, at-least-once, and why exactly-once is hard everywhere

coreintermediate

This is the general version of the delivery-guarantee trade-off that shows up in every distributed messaging system, not just Kafka: at-most-once may silently lose work, at-least-once may silently duplicate it, and exactly-once — never losing and never duplicating — is fundamentally hard to guarantee once a message crosses any network boundary, because the sender can never be certain the receiver's acknowledgment wasn't lost after the work actually succeeded.

Think of it as

Imagine mailing a signed contract and needing confirmation it arrived. At-most-once is sending it once and never checking — if it's lost in the mail, you never know and never resend. At-least-once is resending it every time you don't get a confirmation — safe against loss, but if the confirmation itself was the thing that got lost (not the contract), the recipient now has two signed copies. Exactly-once would mean somehow guaranteeing one copy arrives no matter what — genuinely hard, because you can't tell "contract lost" apart from "confirmation lost" from where you're standing.

What we're doing: Show the exact ambiguity that makes true exactly-once delivery hard — a lost acknowledgment looks identical to a lost message.

ack-ambiguity.txttext

Service A calls Service B to process a payment,
expecting an HTTP 200 acknowledgment.

Scenario 1 (message lost):
  A sends the request. It never reaches B (network
  drop). A sees a timeout — no response.

Scenario 2 (ack lost):
  A sends the request. B receives it, processes the
  payment successfully, sends back 200 OK. The
  RESPONSE is lost on the way back. A sees the exact
  same timeout — no response.

From A's side, scenario 1 and scenario 2 are
INDISTINGUISHABLE — both show up as "no response."
If A retries (correct for scenario 1), it risks
double-processing the payment (wrong for scenario 2)
unless B's processing is itself idempotent.

4: This is the case retrying correctly recovers from — the message never arrived.
15: This is the ambiguity itself: two completely different real outcomes produce the exact same observation at the caller.

Why this works: This ambiguity is not a bug in any particular system — it is a fundamental property of any request/response over an unreliable network, which is why "exactly-once, guaranteed by the delivery mechanism alone" is not achievable in general, only approximated via idempotency on the receiving side.

Believing a specific technology choice (a particular broker, a particular protocol) solves exactly-once for you

Wrong

text

"We switched to [some messaging system]
because it guarantees exactly-once delivery —
we don't need to worry about duplicates
anymore."

Better

text

"No messaging system delivers true exactly-
once across an arbitrary network boundary —
some offer exactly-once semantics WITHIN their
own internal pipeline (see Kafka, section 28),
but any external side effect still needs its
own idempotency to be safe against duplicate
delivery."

What you see: A duplicate side effect (double charge, duplicate email, duplicate order) occurs in a system built on a messaging technology marketed as "exactly-once," and the team is caught off guard because they treated the marketing claim as covering their entire pipeline rather than the specific internal boundary it actually applies to.

Why: The fundamental ack-ambiguity problem applies at every network boundary a message crosses — a technology can only close that gap for hops it fully controls both ends of (like Kafka producing and consuming within itself); the moment a message reaches something the technology doesn't control (an external API, a different system), the ambiguity is back.

Indistinguishable from A's side: both are a timeout

Message lost

+Request never reaches B (network drop)
+B never processes the payment
+A sees a timeout — no response

Ack lost

−B receives it, processes it, sends 200 OK
−The response is lost on the way back
−A sees the exact same timeout

The three semantics, generalized across any distributed messaging system

The three semantics, generalized across any distributed messaging system
Semantic	Can lose?	Can duplicate?	How achieved
At-most-once	Yes	No	Send/process without retry or confirmation
At-least-once	No	Yes	Retry until acknowledged
Exactly-once (effective)	No	No (in effect)	At-least-once delivery + idempotent processing

Remember: At-most-once can silently lose work; at-least-once can silently duplicate it; true exactly-once is fundamentally hard because a sender can never tell "message lost" apart from "acknowledgment lost." The practical target is at-least-once delivery plus idempotent processing.

Designing consumers to be idempotent

coreintermediate

An idempotent operation produces the same end result no matter how many times it is applied — calling it once or five times with the same input leaves the system in the identical final state. Since at-least-once delivery makes duplicate messages a normal, expected occurrence rather than a rare edge case, a consumer that will receive messages under at-least-once semantics needs to be designed idempotent from the start, not patched to handle duplicates after a production incident reveals the gap.

Think of it as

An idempotent light switch is one already labeled "ON" — flipping it to "ON" again changes nothing, no matter how many times you do it. A non-idempotent dimmer that "increases brightness by 10%" is different — press it five times by accident (five duplicate deliveries) and the room is far brighter than intended. Idempotent consumer design means building every switch to behave like the first kind, even when the underlying action would naturally behave like the second.

text

-- non-idempotent: relative change
UPDATE accounts SET balance = balance - 10 WHERE id=1;

-- idempotent: absolute set, or guarded by a
-- unique/already-processed check
UPDATE accounts SET balance = 90 WHERE id=1;

What we're doing: Redesign a naturally non-idempotent "deduct inventory" consumer into an idempotent one.

idempotent-inventory-deduction.txttext

Non-idempotent version:
  on "order_placed" message:
    UPDATE inventory SET stock = stock - 1
      WHERE product_id = :id;
  -- a duplicate delivery of the same order_placed
  -- message deducts stock TWICE for one real order.

Idempotent version:
  on "order_placed" message (with order_id):
    INSERT INTO processed_orders (order_id)
      VALUES (:order_id)
      ON CONFLICT (order_id) DO NOTHING
      RETURNING order_id;
    -- if this insert returned no row, this order_id
    -- was already processed — skip the deduction.
    -- if it returned a row, this is genuinely the
    -- first time — proceed:
    UPDATE inventory SET stock = stock - 1
      WHERE product_id = :id;

5: This is the actual bug — the operation itself (a relative decrement) is not idempotent, so a duplicate delivery directly causes a duplicate deduction.
10: The uniqueness constraint on order_id is the mechanism that makes the second, duplicate delivery a safe no-op instead of a repeat deduction.

Why this works: This pattern — a "have I already processed this ID" check backed by a real database constraint, guarding a non-idempotent operation — is the standard way to make an inherently non-idempotent action safe under at-least-once delivery.

Assuming a message having a unique ID automatically makes processing it idempotent

Wrong

text

"Every message has a unique message_id, so
duplicates aren't a problem — we can tell them
apart."

Better

text

"Every message having a unique ID is necessary
but not sufficient — the CONSUMER has to
actually check that ID against what it has
already processed (e.g. via a unique
constraint or a processed-IDs table) before
acting. An ID nobody checks provides zero
protection on its own."

What you see: A production incident traces back to duplicate processing of a message that, on inspection, did have a unique ID all along — the ID existed, but nothing in the consumer's code path ever looked at it before performing the (non-idempotent) action.

Why: A unique ID is a necessary building block for idempotency, but idempotency itself requires the consumer to actively use that ID to detect and skip repeat processing — simply having a unique field present in the message does nothing by itself.

Idempotent consumer: guard the deduction with a uniqueness check

order_placed message

may be a duplicate

INSERT processed_orders

ON CONFLICT DO NOTHING

Skip

already processed

Deduct inventory

genuinely first time

Naturally idempotent vs naturally non-idempotent operations

Naturally idempotent vs naturally non-idempotent operations
Operation	Idempotent?	Why
SET balance = 90	Yes	Same absolute value every time, regardless of repeat count
balance -= 10	No	Each repeat changes the result further
INSERT with a unique constraint on order_id	Yes (effectively)	A duplicate insert fails harmlessly at the database level
INSERT with no uniqueness constraint	No	Each repeat creates a new, duplicate row
Mark status = "shipped"	Yes	Setting the same status again changes nothing further
Send an email	No (by default)	Each call sends another email — needs an explicit dedup mechanism

Remember: Idempotency is a property of the operation, not the message — a unique message ID does nothing unless the consumer actively checks it, atomically, against what has already been processed. Design for it from the start; duplicates under at-least-once delivery are normal, not rare.

Implementing idempotency concretely

The five mechanisms that turn idempotent design into working code.

Idempotency keys, unique constraints, dedup tables and outbox/inbox

coreintermediate

These are the five concrete mechanisms that actually implement idempotent processing in practice. An idempotency key is a client-generated unique identifier for one logical operation, checked before acting. A unique constraint lets the database itself reject a duplicate. A deduplication table explicitly records processed IDs. A state check verifies current status before acting (skip if already done). The transactional outbox/inbox pattern solves the specific problem of keeping a database write and a message publish (or a message receipt and processing) atomic with each other.

Think of it as

An idempotency key is a claim ticket a customer keeps reusing for retries of the same order, so the shop recognizes "this again" instead of ringing up a new sale. A unique constraint is the shop's till physically refusing to log two sales with the same ticket number. A deduplication table is a notebook the clerk keeps of ticket numbers already served. A state check is glancing at the order board to see it's already marked "done." The outbox pattern is writing the receipt and the ticket stub in the same carbon-copy stroke, so one can never exist without the other.

text

-- idempotency key on an API request
POST /charges
Idempotency-Key: a1b2c3-retry-safe-key
{ "amount": 1000, "customer": "cus_1" }

-- outbox: one local transaction, two tables
BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox_events (...) VALUES (...);
COMMIT;
-- a separate publisher process reads outbox_events
-- and publishes them, marking each as sent

What we're doing: Show the transactional outbox pattern solving the dual-write problem for an order-placement flow.

transactional-outbox.txttext

Naive (dual-write) version:
  BEGIN;
    INSERT INTO orders (...) VALUES (...);
  COMMIT;
  publish_message("order_created", order_id);
  -- if the process crashes between COMMIT and
  -- publish_message, the order exists but the event
  -- NEVER gets published — other services never
  -- learn the order happened.

Outbox version:
  BEGIN;
    INSERT INTO orders (...) VALUES (...);
    INSERT INTO outbox_events (event_type, payload)
      VALUES ('order_created', ...);
  COMMIT;
  -- both inserts succeed or both roll back together
  -- — no window where one happened and not the other.

  -- Separately, a poller/CDC process reads
  -- unpublished outbox_events rows and publishes
  -- them, marking each as sent once confirmed.

5: This is the dual-write gap: two independent operations (a DB commit, a message publish) with no atomicity between them.
14: Both inserts are now in the SAME transaction — there is no possible state where the order exists but its event doesn't.

Why this works: The dual-write problem is a real, common source of silent data-consistency bugs — a crash at exactly the wrong millisecond between two independent operations is rare per-attempt but not rare at scale, and the outbox pattern removes the gap entirely by making both writes part of one atomic local transaction.

Generating a new idempotency key on every retry instead of reusing one per logical operation

Wrong

text

def charge_customer(amount):
    key = generate_uuid()  -- NEW key every call,
                             -- including retries
    api.post("/charges",
             headers={"Idempotency-Key": key},
             json={"amount": amount})

Better

text

def charge_customer(order_id, amount):
    key = f"charge-{order_id}"  -- SAME key for
                                  -- every retry of
                                  -- this specific
                                  -- logical charge
    api.post("/charges",
             headers={"Idempotency-Key": key},
             json={"amount": amount})

What you see: A customer is charged multiple times for what was meant to be a single retried request — because each retry generated a fresh idempotency key, so the receiving system saw what looked like several genuinely distinct charge requests instead of one request retried several times.

Why: An idempotency key only works if it identifies the logical operation, not the individual attempt — generating a new key per retry defeats the entire mechanism, since the receiver has no way to recognize a retry as "the same request" if every attempt claims to be a new one.

Five idempotency mechanisms

Idempotency key

client-generated, reused per retry

Unique constraint

DB rejects a duplicate insert

Dedup table

explicit processed-IDs record

State check

skip if already done

Outbox/inbox

DB write + publish, atomically

Five mechanisms and what each is naturally good for

Five mechanisms and what each is naturally good for
Mechanism	Best fit	Key property
Idempotency key	Client-initiated operations (API requests) that may be retried	Caller controls the key, reused across retries of the same logical operation
Unique constraint	Operations that map to "insert exactly one row"	Database enforces it atomically, no application race condition
Deduplication table	Consumer-side message processing	Explicit, inspectable record of what has been handled
State check	Operations with a natural status field	No extra table needed — the existing state IS the check
Outbox/inbox	Keeping a DB write and a message atomic with each other	Solves the dual-write problem specifically

Remember: Idempotency key for client-retried operations, unique constraint for "insert exactly once," a dedup table or state check on the consumer side, and the outbox/inbox pattern specifically for keeping a database write and a message atomic with each other.