beginnerintermediateadvanced

AWS · Section 63

API Reliability and Idempotency on AWS

Level: advanced
Read: 35 min
Concepts: 3

Every idempotency requirement traces back to one fact: a caller that receives no response cannot tell whether the request was lost, the response was lost, or the work is still running. AWS defines an idempotent operation as one that can be retransmitted or retried with no additional side effects, and its preferred implementation is a unique caller-provided request identifier written into the API contract — EC2 exposes this as ClientToken. Doing it properly has three parts that are easy to miss: the token record and the mutation must be one atomic, consistent, isolated and durable operation, so check-then-act loses the race between two concurrent retries; the retry must receive a semantically equivalent response rather than an error, or the client cannot learn what happened; and the request history must outlive the resource, because a late retry can arrive after the thing it created has been deleted. Five mechanisms implement it, and the choice is usually made by what already identifies the work. A natural key means the storage engine can enforce uniqueness atomically — DynamoDB's attribute_not_exists rejects a second create with "The conditional request failed", and a deterministic S3 key replaces rather than duplicates. Only when nothing natural exists does the caller need to supply a token. SQS FIFO deduplication is narrower than it looks: it prevents a duplicate entering the queue when SendMessage is retried within a five-minute interval, which is producer-side and time-bounded and leaves the consumer's obligation entirely untouched. That is the general shape. Exactly-once delivery is not on offer from any distributed system; exactly-once effect is something you build, and it is the only guarantee actually available.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

The ambiguous timeout, not the retry, is the underlying problem.
Recording the token and doing the work must be one atomic operation.
A retry gets the stored response, not an error the client cannot use.
A natural key lets the storage engine enforce idempotency for free.
Delivery is at least once everywhere; only the effect can be exactly once.

What you will be able to do

Explain why retries are unavoidable and what makes them safe
Implement an idempotent create endpoint, including the retention requirement
Choose between a key, a conditional write, deduplication and durable state
Replace a distributed lock with a conditional write where that is correct
State exactly what an AWS delivery guarantee covers, and what it leaves to you

From an ambiguous timeout to an exactly-once outcome

An ambiguous timeout

lost request or lost response — indistinguishable

Make the retry safe

client key, atomic record, stored response

Pick the mechanism

conditional write, unique key, dedup, durable state

Know what is not guaranteed

delivery is at least once, always

Exactly-once effect

the only guarantee actually available

API Reliability and Idempotency on AWS

Why the ambiguous timeout forces idempotency, the five mechanisms that provide it and how to choose between them, and what AWS delivery guarantees do and do not cover.

What Idempotency Actually Means

coreadvanced

An idempotent operation can be retransmitted or retried with no additional side effects. Running it twice leaves the system in the same state as running it once, and the caller gets the same answer either way. That property is what makes retries safe — and without it, every timeout becomes a choice between losing work and duplicating it.

Think of it as

The problem is not the retry; it is the ambiguous timeout. When a caller does not get a response, it cannot tell whether the operation succeeded, failed, or is still running. Idempotency removes the need to know: retrying is correct in all three cases.

What we're doing: Make a create endpoint safe to retry, and see what "semantically equivalent response" requires.

idempotent-create.txttext

THE CONTRACT — the client supplies the key
  POST /orders
  Idempotency-Key: 6f0c1e3a-…    ← generated by the CLIENT, once,
                                    and reused on every retry

  Generating it on the server defeats the purpose: a retry would
  arrive with a new key and be treated as a new request.

THE HANDLER — record and mutate in one atomic step
  BEGIN
    INSERT INTO idempotency (key, request_hash, status)
    VALUES (:key, :hash, 'in_progress')     -- unique on key
    INSERT INTO orders (…) RETURNING id
    UPDATE idempotency SET status='done', response=:body
  COMMIT

  One transaction. If the process dies anywhere inside it, no
  partial state survives — which is the ACID requirement AWS
  states for combining the token record with the mutation.

THE RETRY — three cases, all handled
  key not present       → first attempt, do the work
  key present, done     → return the STORED response, unchanged
  key present, running  → return 409, tell the client to wait

RETENTION — longer than you expect
  A late retry can arrive after the order has been cancelled and
  deleted. The idempotency record has to outlive the resource, or
  that retry creates a second order.

1: Client-generated is the whole point, and it is the detail most often reversed.
8: The uniqueness constraint on the key is what makes two concurrent retries race safely — one insert wins.
15: Returning the stored response, not a fresh one, is what "semantically equivalent" means in practice.
21: AWS calls this out directly: request history must be kept beyond the lifetime of the resource.

Why this works: Idempotency is usually described as "do not do it twice", which understates it. The operation also has to answer the second caller correctly, atomically, and long after the resource may have gone. Those three requirements — stored response, single transaction, retention beyond resource lifetime — are what separate a real implementation from one that works only in testing.

Checking for an existing record before writing, in two steps

Wrong

text

if not exists(key):        # step 1: read
    create_order()         # step 2: write
    save(key)              # step 3: another write

Better

text

# One atomic conditional write. The database enforces uniqueness.
INSERT INTO idempotency (key, …) VALUES (:key, …)
  -- unique violation → someone else got here first

What you see: Two retries arriving milliseconds apart both pass the existence check, and both create an order — the exact failure the key was added to prevent.

Why: Check-then-act is not atomic, so two concurrent callers can both observe "not present" before either writes. AWS states the requirement precisely: recording the token and the mutating operations must together meet the ACID properties. A conditional write or a unique constraint pushes that guarantee into the storage layer, where it actually holds.

The ambiguous timeout, and what it forces

The client sees one thing — no response — in three completely different situations. Idempotency is what makes the same next action correct in all three.

Where the ambiguous timeout shows up

Where the ambiguous timeout shows up
Place	Why a retry happens	What a duplicate costs
A public API	The client, a proxy, or a mobile network retries	Two orders, two accounts, two of whatever was created
A payment operation	The gateway times out; the caller retries	A customer charged twice — the most expensive duplicate there is
An SQS consumer	The visibility timeout expires before the handler finishes	The message is processed again by another worker
An asynchronous Lambda	Lambda retries the invocation on error	The handler runs twice for one event
An event source mapping	A batch is redelivered after a partial failure	Every record in the batch is reprocessed, not just the failed one
A distributed workflow	A step is retried after a transient error	That step's effect is applied twice mid-workflow

Together

text

# Idempotent by nature, and not — the difference is the verb
PUT /orders/8821  {status: "shipped"}   → idempotent: setting a
                                          value to the same value
                                          twice changes nothing

POST /orders      {items: […]}          → NOT idempotent: creating
                                          twice creates two things

POST /accounts/91/balance/increment 500 → NOT idempotent: the
                                          second call adds again

# Absolute assignment is naturally safe. Creation and relative
# change are not, and those are the ones needing a token.

Remember: An idempotent operation can be retried with no additional side effects, because the client cannot tell a lost response from a lost request. Take a client-supplied key, record it and perform the mutation in one atomic transaction, return the same stored response to every retry, and keep the record longer than the resource itself lives.

The Mechanisms, and Which One to Reach For

coreadvanced

There are five ways to make an operation safe to repeat, and they are not interchangeable. An idempotency key identifies the request; a conditional write makes the first attempt win atomically; deduplication filters repeats within a window; a unique constraint lets the database enforce it; durable state remembers what was already done. Most real designs use two of them together.

Think of it as

Ask what already uniquely identifies this work. If the domain gives you a natural key — an order id, an S3 object key, an invoice number — the storage layer can enforce uniqueness for free. Only when nothing natural exists do you need the caller to supply a token, and that is a contract change.

What we're doing: Choose the mechanism for four operations, and say why each other option is wrong.

choosing.txttext

1. "Create an order from a mobile app"
   → Idempotency key in the API contract.
   Not a natural key: the client has nothing unique before the
   server assigns an id. Not deduplication: a mobile retry can
   arrive an hour later, well outside any window. The key is
   generated once by the client and reused on every attempt.

2. "Claim a job from a queue so only one worker runs it"
   → Conditional write.
   UpdateItem with ConditionExpression status = 'pending'.
   Two workers race, one wins atomically, the loser sees the
   condition fail and exits cleanly. No coordination service,
   no lock, no lease to expire.

3. "Write the processed PDF for job j-8821"
   → Deterministic S3 key.
   processed/t-14/j-8821/report.pdf. Writing it twice replaces
   rather than duplicates, so a worker that dies after writing
   but before acknowledging is harmless.

4. "The publisher retried SendMessage after a timeout"
   → SQS FIFO deduplication.
   Within the 5-minute interval this is free. Beyond it, the
   consumer's own idempotency is what covers you — which is why
   step 2 exists even with step 4 in place.

1: The absence of a natural key is what forces a contract change; that is the test for when a token is genuinely needed.
8: A conditional write replaces a distributed lock in most job-claiming designs, and cannot leak a lease.
15: Determinism in the output key is the cheapest idempotency available — it requires no extra storage at all.
21: The bounded window is the reason producer-side deduplication is never the whole answer.

Why this works: Choosing badly here is expensive in both directions: a token where a natural key exists adds a table and a contract change for nothing, and deduplication where durable state is needed silently stops working once the window closes. Reading the operation for what already identifies it, and how late a retry can arrive, picks the mechanism in one step.

Using a distributed lock where a conditional write would do

Wrong

text

# Acquire a Redis lock on job_id, do the work, release it.
# If the worker dies, the lock expires — and a second worker
# starts while the first may still be running.

Better

text

# Conditional write on the job record:
#   UpdateItem SET status='processing'
#     ConditionExpression: status = 'pending'
# Atomic, durable, and it cannot expire mid-work.

What you see: A worker pauses long enough for its lock to expire, a second worker starts the same job, and both write results — with no error anywhere.

Why: A lock is a lease that must be held for the duration of the work, which makes correctness depend on wall-clock timing and on a component whose failure mode is granting the lock twice. A conditional write records the decision durably at the moment it is made, so the outcome does not depend on how long the work takes.

Five mechanisms, and where each one belongs

The choice is usually made for you by what identifies the work: a natural key means the storage layer can enforce it, and only the absence of one requires a caller-supplied token.

The five mechanisms, side by side

The five mechanisms, side by side
Mechanism	Enforced by	Window	Best for
Idempotency key	Your code plus a unique index	As long as you retain it	Creates over a public API, where no natural key exists
Conditional write	The database, atomically	As long as the item exists	DynamoDB writes, claims, and optimistic concurrency
Unique constraint	The database, atomically	As long as the row exists	Relational creates with a natural key
SQS FIFO deduplication	SQS	5-minute interval	Producer-side retries of the same send
Deterministic object key	S3 replace-on-write	Unbounded	File and artifact processing pipelines
Durable state	Your storage	Whatever you choose	Anything retried after a bounded window closes

Together

text

# DynamoDB: create-once, enforced by the engine
aws dynamodb put-item \
  --table-name Orders \
  --item file://order.json \
  --condition-expression "attribute_not_exists(Id)"

# When the condition is false, DynamoDB returns:
#   The conditional request failed
# That error is the success case for a retry — it means the item
# already exists, so the work is already done.

Producer-side and consumer-side are different problems

Producer-side and consumer-side are different problems
Aspect	Producer-side duplicate	Consumer-side duplicate
Cause	The same message is sent twice after a timeout	One message is delivered or processed more than once
Fixed by	SQS FIFO deduplication, within 5 minutes	An idempotent handler — nothing else
Bounded?	Yes, by the deduplication interval	No — a redelivery can happen at any time
Standard queues	No deduplication at all	Same requirement: the handler must be idempotent
Common error	Assuming FIFO makes the consumer safe too	Assuming a visibility timeout prevents reprocessing

Together

text

# Both, together, for one job
PRODUCER  SQS FIFO with MessageDeduplicationId = job_id
          → a retried SendMessage within 5 minutes adds nothing

CONSUMER  conditional claim on the same job_id
          → a redelivery hours later still processes once

# Neither one alone is sufficient: the first covers the send,
# the second covers everything after it.

Remember: Let the storage engine enforce it whenever a natural key exists — a conditional write or unique constraint is atomic and free. Take a client-supplied idempotency key only when nothing natural identifies the request. FIFO deduplication covers producer retries within five minutes and never covers the consumer, which always needs its own idempotency.

Never Assume Exactly-Once

coreadvanced

Exactly-once delivery is not something a distributed system can offer, because a sender cannot tell a lost message from a lost acknowledgement. What can be built is exactly-once effect: at-least-once delivery plus a handler that produces the same result no matter how many times it runs. Every design that claims otherwise is relying on something being written down somewhere else.

Think of it as

Separate delivery from effect. Delivery is at least once and you do not control it. Effect is exactly once and you do control it. Every guarantee a service does offer — FIFO deduplication, a conditional write — narrows one specific window, and knowing which window is the whole skill.

What we're doing: Show the sequence where a duplicate happens with nothing broken.

nothing-failed.txttext

No component fails in this sequence. There is no bug. A payment
is still taken twice.

  t+0.0s  Worker A receives message m-4417 (visibility 30s)
  t+0.1s  Worker A calls the payment provider
  t+28.0s Provider is slow — still no response
  t+30.0s Visibility timeout expires. SQS makes m-4417 visible
          again, exactly as documented.
  t+30.2s Worker B receives m-4417 and calls the provider
  t+31.0s Provider responds to Worker A: charged
  t+33.0s Provider responds to Worker B: charged again

Everything here behaved correctly:
  - SQS honoured the visibility timeout it was configured with
  - the provider processed two well-formed requests
  - both workers did exactly what they were written to do

WHAT WOULD HAVE PREVENTED IT
  A conditional claim before the call:
    UpdateItem(pk=m-4417, SET status='charging',
               ConditionExpression: attribute_not_exists(status))
  Worker B's condition fails, it stops, and the second charge
  never happens.

  Not a longer visibility timeout: that moves the boundary and
  removes nothing. There is always a duration that exceeds it.

1: This is the important framing — duplicate processing is usually the correct behaviour of correctly configured components.
9: The visibility timeout expiring is a feature working as designed; it is what stops a dead worker from stranding a message forever.
16: Tuning the timeout is the instinctive fix and it never closes the gap, only moves it.

Why this works: Teams look for a broken component after a duplicate and do not find one, then conclude it was a rare anomaly. It was not: it is the documented behaviour of at-least-once delivery meeting a slow dependency. Accepting that changes the fix from tuning timeouts to making the effect idempotent, which is the only thing that actually closes the window.

Raising the visibility timeout to stop duplicates

Wrong

text

# "Processing takes about 30s and we saw duplicates,
# so set the visibility timeout to 15 minutes."

Better

text

# Size the visibility timeout to the expected processing time,
# extend it explicitly for long work, and make the handler
# idempotent — which is what actually prevents the duplicate.

What you see: Duplicates become rarer and do not stop, while a worker that dies now leaves its message invisible for fifteen minutes — turning a fast recovery into a long stall.

Why: The visibility timeout trades one failure for another: too short redelivers work that is still running, too long delays recovery after a crash. Neither setting makes processing exactly once, because the message becomes visible again whenever the timeout expires — by design, so that a dead worker cannot hold a message forever.

What each layer actually guarantees

Exactly-once delivery

Not available. A sender cannot tell a lost message from a lost acknowledgement, so it must retry, so duplicates exist.

At-least-once delivery

What SQS, SNS, EventBridge, S3 notifications and Lambda event sources actually provide. Assume it everywhere.

Bounded deduplication

FIFO removes duplicate sends within a 5-minute interval. Narrow, useful, and not a processing guarantee.

Exactly-once effect

What you build: a conditional write, a unique key, or stored durable state. This is the only layer under your control.

What each AWS service actually promises

What each AWS service actually promises
Service	Delivery	Deduplication	What you still owe
SQS standard	At least once	None	An idempotent consumer
SQS FIFO	At least once to the consumer	Producer sends, within a 5-minute interval	An idempotent consumer, still
SNS	At least once	None	Idempotency in every subscriber
EventBridge	At least once	None	Idempotency in every target
S3 event notifications	At least once	None	Derive identity from the object key
Lambda, asynchronous	Retried by the service on error	None	An idempotent handler
Lambda event source mapping	Batches can be redelivered	None	Per-record idempotency, not per-batch
DynamoDB conditional write	Not applicable	Not applicable	Nothing — this is the mechanism that gives the guarantee

Together

text

# Read the column that matters: "what you still owe" is never
# empty except in the last row. That is the point of the table.
#
# Choosing FIFO over standard changes the DEDUPLICATION column.
# It does not change the OBLIGATION column, which is where every
# duplicate-processing incident actually comes from.

Remember: Delivery is at least once and you do not control it; effect is exactly once and you do. Every AWS guarantee — FIFO deduplication in particular — narrows one specific, documented window and leaves the consumer's obligation untouched. Write down what each guarantee does not cover, and put a conditional write there.