beginnerintermediateadvanced

AWS · Section 26

Messaging — SQS

Level: intermediate
Read: 32 min
Concepts: 5

SQS decouples producers from consumers with a durable queue in between. This section covers the core vocabulary — visibility timeout, long polling, standard vs FIFO queues, message groups, and deduplication — then dead-letter queues and redrive for messages that keep failing, why at-least-once delivery makes idempotent consumers mandatory rather than optional, how to size and extend visibility timeout for real workloads, and how to bound worker concurrency so a queue-driven consumer never overwhelms whatever capacity-limited system sits behind it.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

A received message is hidden for the visibility timeout and reappears if not deleted in time — size it to the job, not the 30-second default.
Standard queues favor throughput with best-effort ordering; FIFO queues guarantee strict order per MessageGroupId at a much lower throughput ceiling.
maxReceiveCount on a redrive policy moves a repeatedly-failing message to a DLQ of the same queue type, instead of retrying it forever.
SQS delivers at-least-once, never exactly-once outside a FIFO dedup window — every consumer needs an idempotency key and a conditional write.
Bounded concurrency (Lambda maximum concurrency, or a fixed worker pool) keeps a queue-driven consumer from overwhelming a downstream dependency with a hard capacity limit.

What you will be able to do

Choose between a standard and a FIFO queue based on whether ordering is a real requirement
Configure a redrive policy and dead-letter queue so failing messages are captured, not retried forever or silently dropped
Write a consumer that is safe to run twice on the same message, using an idempotency key and a conditional write
Size and, when needed, extend a visibility timeout to match real processing time
Bound a queue-driven worker's concurrency to what its downstream dependency can actually sustain

From queues and delivery guarantees to a bounded worker architecture

Queues, visibility timeout, standard vs FIFO

DLQ, redrive, idempotent consumers

At-least-once delivery, duplicates

Sizing/extending visibility timeout

Bounded worker architectures

Messaging — SQS

Queue vocabulary and delivery modes, dead-letter queues and idempotent consumers, at-least-once delivery, visibility timeout sizing, and bounded worker architectures.

SQS Queues, Visibility Timeout, Long Polling, Standard vs FIFO

coreintermediate

An SQS queue holds messages until a consumer processes and deletes them. When a consumer receives a message, it stays in the queue but becomes invisible for the visibility timeout — if the consumer does not delete it in time, it reappears for another consumer. Long polling waits up to 20 seconds for a message instead of returning empty immediately, cutting wasted API calls. A standard queue has unlimited throughput but only best-effort ordering; a FIFO queue guarantees strict order within a message group, at a much lower throughput ceiling.

Think of it as

A message is like a claim ticket at a coat check: receiving it does not remove it from the rack, it just flips a "reserved" flag (the visibility timeout) so nobody else grabs the same coat. If you never come back to claim it (delete it), the flag clears and someone else can take it. Long polling is the difference between an attendant who checks the rack once and shrugs if it is empty, versus one who watches the door for up to 20 seconds before giving up.

What we're doing: See why a consumer that takes longer than the visibility timeout causes the same message to be processed twice.

visibility-timeout-race.txttext

Queue visibility timeout: 30s
Consumer A receives order-4471, starts a 45s image-resize job
  t=30s → visibility timeout expires, order-4471 becomes visible again
  t=31s → Consumer B receives the SAME order-4471, starts processing it too
  t=45s → Consumer A finishes, calls DeleteMessage — too late, B is mid-flight
→ order-4471 gets processed twice

1: The visibility timeout is shorter than the actual processing time — the root cause.
4: A second consumer receiving the reappeared message is expected SQS behavior, not a bug — the queue has no way to know Consumer A is still working.

Why this works: SQS is at-least-once, not exactly-once (except FIFO's within-group exactly-once processing guarantee) — a visibility timeout shorter than real processing time is one of the most common ways a duplicate delivery actually happens in production, and it is entirely avoidable by sizing the timeout to the job.

Leaving the visibility timeout at the 30-second default for a slow job

Wrong

text

# Queue created with default settings; consumer does a 2-minute video
# transcode per message

Better

text

# Set visibility timeout to comfortably exceed the expected processing
# time (or use a heartbeat that calls ChangeMessageVisibility periodically)

What you see: The same message gets processed by more than one consumer concurrently, and CloudWatch shows ApproximateReceiveCount climbing on messages that were never actually stuck.

Why: The default (30s) is a generic starting point, not a recommendation for any specific job — a job that runs longer than the timeout guarantees a duplicate receive, regardless of how reliable the consumer code is.

Standard vs FIFO queue

Standard

+Nearly unlimited throughput per API action
+Best-effort ordering — messages can arrive out of order
+At-least-once delivery — duplicates are possible

FIFO

−300 TPS per API action by default (3,000 msg/sec batched)
−Strict order guaranteed within a MessageGroupId
−Deduplication window (default 5 min) prevents duplicate sends

Remember: Visibility timeout hides a received message so only one consumer works it; it reappears if not deleted in time — size it to the job. Long polling (up to 20s) avoids wasted empty responses. Standard = unlimited throughput, best-effort order; FIFO = strict order per MessageGroupId, ~300 TPS default ceiling, dedup window prevents repeat sends.

Dead-Letter Queues, Redrive, Retries, and Idempotent Consumers

coreintermediate

A dead-letter queue (DLQ) is a separate queue that catches messages a consumer has failed to process too many times. A redrive policy sets maxReceiveCount — the number of receives before a message moves to the DLQ — and the DLQ must be the same queue type (standard or FIFO) as its source. Because SQS delivers at-least-once, a consumer can receive the same message more than once even without failures, so it must be idempotent: processing the same message twice must produce the same result as processing it once.

Think of it as

Think of maxReceiveCount as "three strikes": each time a message is received and not deleted, that counts as a strike. On the strike after maxReceiveCount, the message is moved to the DLQ instead of going back to the main queue — a holding pen for messages nothing can currently process, so you can inspect them without them clogging retries for everyone else.

What we're doing: See why a consumer that is not idempotent double-charges a customer on an ordinary at-least-once redelivery — no DLQ or failure required.

non-idempotent-charge.txttext

Consumer receives payment message, msg_id=pay_9931
  charge_card(amount=42.00)   # succeeds
  DeleteMessage call times out on the network before it reaches SQS
→ SQS never learns the message was handled — visibility timeout expires
Consumer (or another instance) receives pay_9931 again
  charge_card(amount=42.00)   # charges the card a second time

2: The business action (the charge) succeeds, but the acknowledgment (DeleteMessage) is what tells SQS the work is done.
5: No error occurred anywhere — a transient network blip on the delete call alone is enough to trigger a legitimate at-least-once redelivery.

Why this works: DeleteMessage failing after successful processing is exactly the gap at-least-once delivery leaves open — the fix is not "make the network more reliable", it is making charge_card safe to call twice for the same msg_id.

Treating "no errors in the logs" as proof a message was processed exactly once

Wrong

text

# No idempotency key check — charge_card() runs unconditionally for
# every message the consumer receives

Better

text

# Before charging: check whether msg_id (or an order ID) was already
# processed (e.g. a conditional write in DynamoDB); skip if so

What you see: A small, hard-to-reproduce rate of duplicate side effects (double charges, duplicate emails, duplicate rows) that never shows up in error logs, because nothing actually errored.

Why: SQS's at-least-once guarantee is a documented, permanent property of the service, not an edge case — any consumer that assumes single delivery will eventually duplicate a side effect, with no error to point at afterward.

A message that keeps failing

Source queue

Consumer fails to delete

ApproximateReceiveCount rises

maxReceiveCount hit → moved to DLQ

Remember: maxReceiveCount on a redrive policy sends a message to the DLQ after N failed receives; the DLQ must match the source queue type and needs a longer retention period. SQS is at-least-once, always — a consumer must be idempotent (track processed IDs) regardless of whether anything ever fails.

At-Least-Once Delivery and Duplicate Message Handling

coreintermediate

SQS guarantees a message is delivered at least once — never zero times — but does not guarantee exactly once for standard queues. A message can be delivered more than once because a consumer's delete request is lost, a visibility timeout expires while work is still in progress, or (for FIFO) a network retry resends the same send request. Handling duplicates is the consumer's job, not the queue's.

Think of it as

At-least-once is a promise about not losing a message, not a promise about delivering it exactly once — those are two different guarantees, and SQS (for standard queues) only makes the first one. Treat every message handler as if it might run twice on the same input, because on a long enough timeline, it will.

What we're doing: See how a conditional write turns a naturally non-idempotent action (incrementing a counter) into one safe to run twice.

idempotent-counter-update.txttext

On receiving message with MessageId = "msg_a1b2":
  PutItem(table=ProcessedMessages, key=msg_a1b2,
           ConditionExpression="attribute_not_exists(id)")
  → if it succeeds: increment the counter, then DeleteMessage
  → if it fails with ConditionalCheckFailedException: skip the increment,
    DeleteMessage anyway (already handled, safe to acknowledge)

2: The conditional write is the atomic "have I seen this before?" check — it either claims the ID or fails, with no race window between check and claim.
3: A second delivery of the same message hits the ConditionalCheckFailedException branch and does nothing further, instead of incrementing the counter again.

Why this works: Checking "have I processed this?" and then writing are two separate steps unless combined atomically — a plain read-then-write has a race window where two concurrent deliveries can both pass the check before either writes.

Using a plain read-then-write check instead of a conditional write

Wrong

text

# if not exists(msg_id in ProcessedMessages):
#     increment_counter()
#     mark_processed(msg_id)   # two round trips, not atomic

Better

text

# PutItem with ConditionExpression="attribute_not_exists(id)" —
# the check and the claim happen in one atomic request

What you see: Under concurrent redelivery (two consumers receive the same message at nearly the same time), both pass the read check before either finishes the write, and the counter is incremented twice anyway.

Why: A separate read and write is not atomic — the race window between them is exactly the case duplicate delivery is likely to hit, since a redelivery is often near-simultaneous with the original delivery still in flight.

A redelivered message, handled idempotently

SQS

Consumer

ProcessedMessages table

1. deliver msg_a1b2 (2nd time)
2. PutItem, condition: not exists
3. ConditionalCheckFailedException
4. skip increment, DeleteMessage anyway

Remember: SQS promises a message is never lost, not that it is delivered exactly once (except within a FIFO queue's dedup window). Duplicates happen from lost deletes, expired visibility timeouts, and producer retries — handle them with an idempotency key and a conditional write, regardless of queue type.

Sizing and Extending Visibility Timeout

standardintermediate

Set the visibility timeout comfortably above the worst-case processing time, not the default 30s. If a job's duration is unpredictable, extend it while work continues by calling ChangeMessageVisibility on a heartbeat instead of guessing one fixed number up front — but the 12-hour ceiling from first receive is absolute and no extension ever resets it.

Think of it as

Size the initial visibility timeout to the typical processing time, with headroom — not the shortest case, not the default. For work whose duration is unpredictable, extend the timeout while processing continues, rather than guessing one fixed number up front.

extend-visibility.shbash

# Extend a specific in-flight message's timeout to 120s from now
aws sqs change-message-visibility \
  --queue-url "$QUEUE_URL" \
  --receipt-handle "$RECEIPT_HANDLE" \
  --visibility-timeout 120

What we're doing: Use a heartbeat to extend visibility timeout for a job whose duration varies.

heartbeat-pattern.txttext

Queue visibility timeout: 60s
Consumer receives a message, starts processing
  every 30s while still working: ChangeMessageVisibility → +60s from now
  on success: DeleteMessage
  on crash: no more heartbeats sent → timeout expires normally → redelivered

2: The heartbeat runs on a fixed interval shorter than the timeout, so the message never actually goes invisible-then-visible while work is still healthy.
4: A crash simply stops the heartbeats — the existing timeout still expires and the message is redelivered normally, with no special crash-handling code needed.

Why this works: A heartbeat converts "guess one number that covers every case" into "keep proving you are still alive" — it handles both a job that finishes in 5 seconds and one that takes 5 minutes without changing the queue's base setting.

Remember: Size the timeout to the expected processing time plus headroom, not the default. For variable-duration work, extend it periodically with ChangeMessageVisibility (a heartbeat) rather than guessing one large fixed value — but the 12-hour ceiling from first receive never resets.

Worker Architectures with Backpressure and Bounded Concurrency

coreadvanced

A worker architecture pulls messages from a queue at a rate the downstream system can actually absorb, instead of processing every message the instant it arrives. Bounded concurrency caps how many messages are worked on at once; backpressure is the mechanism that lets the whole pipeline slow down gracefully — via the queue itself acting as a buffer — instead of a downstream dependency being overwhelmed.

Think of it as

The queue is a shock absorber between producers and a downstream system with a fixed capacity — a database, a third-party API, a fixed pool of workers. Bounding concurrency is choosing how wide the pipe out of that shock absorber is; the queue depth (backlog) is the pressure gauge showing whether the pipe is wide enough for the current load.

What we're doing: Cap a Lambda-based SQS consumer so it never exceeds a downstream database's connection limit.

bound-lambda-concurrency.shbash

# Downstream RDS instance allows ~80 concurrent connections; other apps
# already use some of that budget, so this worker gets 50.
aws lambda put-function-concurrency \
  --function-name process-orders --reserved-concurrent-executions 50

aws lambda update-event-source-mapping \
  --uuid "$ESM_UUID" --scaling-config '{"MaximumConcurrency":50}'

2: Reserved concurrency on the function is the hard ceiling everything else must stay under — it is what actually protects the function's own account-wide concurrency budget.
5: Maximum concurrency on the event source is what actually throttles the SQS-driven invocation rate; it must be set at or below the function's reserved concurrency, or Lambda can throttle unpredictably.

Why this works: Without a cap, default scaling can ramp to 1,250 concurrent invocations for one event source — each opening its own database connection — and exhaust a connection pool sized for a fraction of that, causing every consumer (not just this one) to start failing.

Leaving Lambda's SQS event source at default (unbounded) scaling for a downstream system with a hard capacity limit

Wrong

text

# No maximum concurrency configured — trusting default scaling to
# "figure it out" for a consumer writing to a fixed-size connection pool

Better

text

# Set maximum concurrency on the event source to a number the
# downstream dependency can actually sustain, with margin for other
# callers of the same resource

What you see: A traffic spike causes Lambda to scale up as designed, which in turn exhausts the database's connection pool — and now every service sharing that database fails, not just the one that scaled.

Why: Unbounded scaling is correct behavior for a queue with no fixed downstream limit, but a shared, fixed-capacity dependency needs an explicit cap — the queue only protects itself from overload, not whatever is behind the worker.

Producer, buffer, bounded workers

Producers (bursty)

SQS queue (buffer)

Bounded worker pool / max concurrency

Downstream capacity (DB, API)

Remember: Bounded concurrency (maximum concurrency, or a fixed-size worker pool) keeps a queue-driven consumer from overwhelming a downstream system that scales less elastically than the queue does. The queue itself provides backpressure by buffering — a growing queue depth under a deliberately fixed worker count is the buffer working as intended, not automatically a fault.