beginnerintermediateadvanced

AWS · Messaging — SQS

Messaging — SQS

Queues, delivery guarantees, dead-letter handling, and bounded worker architectures

Visibility timeout: 30s default, 12h maxsize it to the job, or extend via a heartbeat
Standard = throughput, FIFO = orderFIFO caps at 300 TPS per action by default
At-least-once, alwaysidempotent consumers are required, not optional
Bound concurrency to downstream capacitythe queue buffers; the worker pool must not overload what is behind it

1.SQS Queues, Visibility Timeout, Long Polling, Standard vs FIFO

An SQS queue holds messages until a consumer processes and deletes them. When a consumer receives a message, it stays in the queue but becomes invisible for the visibility timeout — if the consumer does not delete it in time, it reappears for another consumer. Long polling waits up to 20 seconds for a message instead of returning empty immediately, cutting wasted API calls. A standard queue has unlimited throughput but only best-effort ordering; a FIFO queue guarantees strict order within a message group, at a much lower throughput ceiling.

Key features

Visibility timeout: default 30 seconds, adjustable per-queue or per-message via ChangeMessageVisibility, hard ceiling 12 hours from first receive
A message not deleted before the visibility timeout expires becomes visible again and can be received by any consumer — this is the retry mechanism
Long polling (WaitTimeSeconds > 0, max 20s) queries all servers and waits for a message; short polling (the default, WaitTimeSeconds = 0) samples a subset and returns immediately even if empty
Standard queue: nearly unlimited throughput, at-least-once delivery, best-effort ordering (messages can arrive out of order or more than once)
FIFO queue: strict ordering within a MessageGroupId, 300 TPS per API action by default (3,000 msg/sec with batching) — MessageGroupId is required on every message
FIFO deduplication: a 5-minute window (default) rejects a repeat of the same MessageDeduplicationId or, with content-based deduplication enabled, the same message body hash

Standard vs FIFO queue

Standard

Nearly unlimited throughput per API action
Best-effort ordering — messages can arrive out of order
At-least-once delivery — duplicates are possible

FIFO

300 TPS per API action by default (3,000 msg/sec batched)
Strict order guaranteed within a MessageGroupId
Deduplication window (default 5 min) prevents duplicate sends

Signature

Visibility timeout (30s default, 12h max) · Long polling (WaitTimeSeconds up to 20s) · Standard (unlimited, best-effort order) vs FIFO (300 TPS default, strict order per group)

Visibility timeout hides a received message so only one consumer works it; it reappears if not deleted in time — size it to the job. Long polling (up to 20s) avoids wasted empty responses. Standard = unlimited throughput, best-effort order; FIFO = strict order per MessageGroupId, ~300 TPS default ceiling, dedup window prevents repeat sends.

2.Dead-Letter Queues, Redrive, Retries, and Idempotent Consumers

A dead-letter queue (DLQ) is a separate queue that catches messages a consumer has failed to process too many times. A redrive policy sets maxReceiveCount — the number of receives before a message moves to the DLQ — and the DLQ must be the same queue type (standard or FIFO) as its source. Because SQS delivers at-least-once, a consumer can receive the same message more than once even without failures, so it must be idempotent: processing the same message twice must produce the same result as processing it once.

Key features

A redrive policy sets maxReceiveCount (1-1,000) on the source queue — after that many receives without deletion, the message moves to the DLQ instead of reappearing
The DLQ must be created first and must match the source queue type: a FIFO source queue requires a FIFO DLQ, a standard source requires a standard DLQ
DLQ redrive (the redrive task feature) moves messages from the DLQ back to the source queue, or to another queue, once the underlying issue is fixed — rather than replaying them by hand
A message's retention period is based on its original enqueue time, unchanged by the move to a DLQ for a standard queue — so a DLQ needs a retention period longer than the source queue's, or messages can expire before anyone looks at them
SQS guarantees at-least-once delivery, never exactly-once, for standard queues — the same message can arrive twice even with zero errors, so idempotency is required, not optional
An idempotent consumer tracks processed message IDs (or a business-level idempotency key) and skips or safely no-ops a repeat, rather than assuming ReceiveMessage only ever returns a message once

A message that keeps failing

deliverstimeout expires,redeliveredafter Nreceives

Source queue

Consumer fails to delete

ApproximateReceiveCount rises

maxReceiveCount hit → moved to DLQ

Signature

Redrive policy: maxReceiveCount (1-1,000) · DLQ type must match source · DLQ redrive moves messages back

maxReceiveCount on a redrive policy sends a message to the DLQ after N failed receives; the DLQ must match the source queue type and needs a longer retention period. SQS is at-least-once, always — a consumer must be idempotent (track processed IDs) regardless of whether anything ever fails.

3.At-Least-Once Delivery and Duplicate Message Handling

SQS guarantees a message is delivered at least once — never zero times — but does not guarantee exactly once for standard queues. A message can be delivered more than once because a consumer's delete request is lost, a visibility timeout expires while work is still in progress, or (for FIFO) a network retry resends the same send request. Handling duplicates is the consumer's job, not the queue's.

Key features

Standard queues: at-least-once delivery — no ordering guarantee and no exactly-once guarantee, duplicates are an expected, documented possibility
FIFO queues: exactly-once processing within the deduplication window, achieved by rejecting a repeat MessageDeduplicationId (or repeat content hash) at send time — this does not remove the need for an idempotent consumer outside that window
Common duplicate causes: DeleteMessage request lost or timed out after successful processing, visibility timeout expiring before a slow consumer finishes, and a producer retrying a SendMessage call that actually succeeded the first time
The standard fix is an idempotency key: derive a stable key from the message (a business ID, or MessageId for SQS-generated IDs) and check/record it in a store with a conditional write before acting
A conditional write (e.g. DynamoDB PutItem with a condition that the key does not already exist) is a common, cheap way to implement the duplicate check atomically
Idempotency matters most for side effects that are not naturally safe to repeat — charging a card, sending an email, incrementing a counter — a duplicate GET-style read has no such problem

A redelivered message, handled idempotently

SQS

Consumer

ProcessedMessages table

1deliver msg_a1b2 (2nd time)
2PutItem, condition: not exists
3ConditionalCheckFailedException
4skip increment, DeleteMessage anyway

Signature

At-least-once (standard) · exactly-once within dedup window (FIFO) · idempotency key + conditional write

SQS promises a message is never lost, not that it is delivered exactly once (except within a FIFO queue's dedup window). Duplicates happen from lost deletes, expired visibility timeouts, and producer retries — handle them with an idempotency key and a conditional write, regardless of queue type.

4.Sizing and Extending Visibility Timeout

Set the visibility timeout comfortably above the worst-case processing time, not the default 30s. If a job's duration is unpredictable, extend it while work continues by calling ChangeMessageVisibility on a heartbeat instead of guessing one fixed number up front — but the 12-hour ceiling from first receive is absolute and no extension ever resets it.

Key features

Starting point: set the timeout to comfortably exceed the maximum expected processing time, not the average — a p50 estimate under-covers the slow tail
ChangeMessageVisibility extends (or shortens) an in-flight message's timeout using its receipt handle, without touching the queue's default setting
A heartbeat pattern — calling ChangeMessageVisibility periodically while still processing — handles variable-duration work better than one large fixed guess
The 12-hour ceiling is absolute and cumulative from the first receive; extending the timeout never resets that clock, so work that can genuinely exceed 12 hours needs breaking into smaller steps (e.g. via Step Functions), not a longer single timeout
Setting VisibilityTimeout to 0 via ChangeMessageVisibility immediately releases a message back to the queue — useful when a consumer decides early it cannot handle a message and wants a retry sooner than waiting out the timeout
Too short a timeout causes duplicate concurrent processing (another consumer picks it up while the first is still working); too long a timeout delays every retry of a message that actually failed fast

Command

extend-visibility.shbash

# Extend a specific in-flight message's timeout to 120s from now
aws sqs change-message-visibility \
  --queue-url "$QUEUE_URL" \
  --receipt-handle "$RECEIPT_HANDLE" \
  --visibility-timeout 120

Size the timeout to the expected processing time plus headroom, not the default. For variable-duration work, extend it periodically with ChangeMessageVisibility (a heartbeat) rather than guessing one large fixed value — but the 12-hour ceiling from first receive never resets.

5.Worker Architectures with Backpressure and Bounded Concurrency

A worker architecture pulls messages from a queue at a rate the downstream system can actually absorb, instead of processing every message the instant it arrives. Bounded concurrency caps how many messages are worked on at once; backpressure is the mechanism that lets the whole pipeline slow down gracefully — via the queue itself acting as a buffer — instead of a downstream dependency being overwhelmed.

Key features

For a Lambda SQS event source, maximum concurrency (2-1,000, set per event source) caps how many concurrent function instances that queue can invoke — independent of the function's own reserved concurrency, which should be set >= the sum of all event sources' maximum concurrency
Default (unbounded) SQS-to-Lambda scaling starts at 5 concurrent invocations and adds up to 300 more per minute, capping at 1,250 for a single event source mapping — enough to overwhelm an under-provisioned downstream dependency if concurrency is left uncapped
For FIFO queues, actual concurrency is capped at whichever is lower: the maximum concurrency setting, or the number of distinct MessageGroupIds currently in flight — more consumers than message groups does not increase throughput
A worker pool with a fixed size (e.g. N long-running consumer processes, each pulling one message at a time) is the equivalent bounded-concurrency pattern outside Lambda — the queue absorbs backlog while the pool stays fixed
Queue depth (ApproximateNumberOfMessagesVisible) rising over time under a fixed worker count is the signal that consumers are slower than producers — the queue is doing its job as a buffer, not necessarily a problem by itself
A batch window (SQS event source batching, up to 5 minutes) trades a small amount of latency for fewer, larger invocations — useful when per-invocation overhead matters more than per-message latency

Producer, buffer, bounded workers

send,any ratepull,capped ratenever exceedscapacity

Producers (bursty)

SQS queue (buffer)

Bounded worker pool / max concurrency

Downstream capacity (DB, API)

Signature

Lambda SQS maximum concurrency: 2-1,000 · default scaling caps at 1,250 · FIFO capped by MessageGroupId count too

Bounded concurrency (maximum concurrency, or a fixed-size worker pool) keeps a queue-driven consumer from overwhelming a downstream system that scales less elastically than the queue does. The queue itself provides backpressure by buffering — a growing queue depth under a deliberately fixed worker count is the buffer working as intended, not automatically a fault.

Spotted a mistake on this page? Tell us.

Read Sheet