beginnerintermediateadvanced

AWS · Section 27

SNS and Event Fan-Out

Level: intermediate
Read: 20 min
Concepts: 3

SNS's vocabulary — topics, subscriptions, filter policies, fan-out, retries, dead-letter queues, and delivery targets — sets up the pattern that uses it in practice: subscribing one SQS queue per consumer to a topic, so a single publish durably reaches every consumer at its own pace, with none of them able to affect the others or the publisher. That decoupling is also the reason to reach for pub/sub instead of a direct synchronous call in the first place — a publisher that does not need a response, or that has more than one downstream service reacting to the same event, gains from never blocking on, or even knowing about, its subscribers.

This section

Fan out Architecture in AWS with SNS + SQS + Lambda + PythonSoumil Shah

What is true here

A subscription, not a topic, carries the filter policy, the retry behaviour, and any dead-letter queue — delivery and delivery failure both happen at the subscription level.
Fan-out is one publish reaching every matching subscription in parallel, each getting its own independent copy.
One SQS queue per consumer is required for true fan-out; consumers sharing a single queue split the message stream instead of each receiving every message.
Retry budgets differ by delivery protocol (AWS-managed endpoints get far more attempts, over far longer, than customer-managed ones) — durability is not uniform across subscriber types without a DLQ.
Pick a direct synchronous call only when the caller needs the result to continue; pick pub/sub when it does not, or when multiple services must react to the same event.

What you will be able to do

Design a topic's subscriptions and filter policies so each subscriber receives only the messages it needs
Set up an SNS + SQS fan-out so multiple consumers process the same event independently and durably
Attach a dead-letter queue where a subscription cannot afford to silently lose a message
Decide when a service-to-service interaction should be a direct synchronous call versus a published event

From the SNS vocabulary to choosing pub/sub over a direct call

Topics, subscriptions, filtering, DLQ

SNS + SQS fan-out pattern

Pub/sub vs direct synchronous calls

The SNS vocabulary (topics, subscriptions, filtering, fan-out, retries, DLQ, delivery targets), the SNS + SQS fan-out pattern for decoupling, and when pub/sub beats a direct synchronous call.

coreintermediate

A topic is a named channel a publisher sends messages to, without knowing who — if anyone — is listening. A subscription attaches one delivery target (SQS, Lambda, HTTP(S), email, SMS, mobile push) to a topic; every subscription gets its own copy of each message. A filter policy on a subscription drops messages that do not match, so one topic can serve subscribers that each want a different subset. Fan-out is what happens when a topic has multiple subscriptions: one publish reaches every matching subscriber in parallel.

Think of it as

A topic is a mailing list, not a mailbox. The publisher drops one message in the list; SNS photocopies it and delivers one copy to every current subscriber, each independently, each on its own retry schedule. Adding or removing a subscriber never touches the publisher's code.

What we're doing: Filter a topic so only high-priority orders reach a paging subscriber, while a logging subscriber still sees every order.

subscription-filter-policy.jsonjson

// Filter policy attached to the "paging" subscription only
{
  "priority": ["high", "urgent"]
}

2: This subscription receives a message only if its "priority" message attribute is "high" or "urgent" — every other publish to the topic is skipped for this subscriber.

Why this works: The filter lives on the subscription, not the topic, so the same topic and the same publish call can serve a narrowly-filtered subscriber (paging) and a see-everything subscriber (logging) at once — the publisher never needs to know either one exists.

Putting routing logic in the publisher instead of a filter policy

Wrong

python

if order["priority"] in ("high", "urgent"):
    sns.publish(TopicArn=PAGING_TOPIC, Message=json.dumps(order))
sns.publish(TopicArn=LOGGING_TOPIC, Message=json.dumps(order))

Better

python

sns.publish(
    TopicArn=ORDERS_TOPIC,
    Message=json.dumps(order),
    MessageAttributes={"priority": {"DataType": "String", "StringValue": order["priority"]}},
)
# Paging subscription carries a filter policy: {"priority": ["high", "urgent"]}
# Logging subscription carries no filter policy — sees every message

What you see: Every new subscriber that wants a different subset of orders requires a code change and a redeploy of the publisher, and the publisher accumulates a growing list of topic ARNs and if/else branches.

Why: A filter policy moves the "who wants this message" decision to the subscriber side, which is what fan-out is for — the publisher publishes once, to one topic, and stays unaware of how many subscribers exist or what they each care about.

One publish, three independent deliveries

Publisher

SNS topic

SQS subscriber

Lambda subscriber

Email subscriber

Remember: Topic = named channel; subscription = one delivery target attached to it, each with its own filter policy and retry schedule. Fan-out is one publish reaching every matching subscriber in parallel. Client-side errors are not retried; server-side errors retry per-protocol (SQS/Lambda: ~23 days; SMTP/SMS/push: ~6 hours) before being discarded — attach a DLQ per subscription to catch what would otherwise be lost.

See also: sns sqs fanout pattern

coreintermediate

SNS pushes; SQS holds. Subscribe several SQS queues to one SNS topic, and every message published reaches every queue as its own durable copy, each consumer polling and processing at its own pace. If one consumer's queue backs up or its worker crashes, the message still sits safely in that queue — the other consumers, and the publisher, are unaffected.

Think of it as

SNS alone is push-only and does not hold a message after a failed delivery beyond its own retry budget. Putting an SQS queue in front of each consumer gives that consumer a durable, poll-based buffer — the topic still fans a single publish out to every subscriber, but each subscriber now owns a queue it can fall behind on, retry from, and scale independently, without the publisher or any other consumer noticing.

What we're doing: Fan an "order placed" event out to a fulfillment consumer and an analytics consumer that process at very different rates.

fanout-setup.shbash

aws sns create-topic --name order-events
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$FULFILLMENT_QUEUE_ARN"
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$ANALYTICS_QUEUE_ARN"
aws sns publish --topic-arn "$TOPIC_ARN" --message '{"order_id": "ord_9021", "total_cents": 4599}'

2: The fulfillment queue subscribes independently — it will hold every order until the fulfillment worker polls it, even if that worker is briefly offline.
3: The analytics queue subscribes to the same topic and gets its own full copy of every message — a slow nightly-batch analytics consumer never blocks or delays fulfillment.

Why this works: One publish call reaches both queues. Fulfillment can process within seconds while analytics polls once an hour — each queue buffers independently, so neither consumer's pace affects the other or the publisher.

Subscribing multiple consumers directly to one shared SQS queue instead of one queue per consumer

Wrong

text

# Fulfillment worker and analytics worker both poll the SAME SQS queue

Better

text

# One SNS topic, TWO separate SQS queues (one per consumer),
# each queue subscribed to the topic independently

What you see: Each message is delivered to only one of the two workers (SQS hands a message to a single consumer, not both) — fulfillment and analytics silently split the order stream instead of both seeing every order.

Why: A shared SQS queue distributes messages across its consumers; it does not duplicate them. Fan-out — every consumer seeing every message — requires one queue per consumer, each subscribed to the topic separately, which is exactly what SNS's topic-to-many-subscriptions model provides.

SNS alone vs SNS + SQS per consumer

SNS alone (push only)

+A down or slow consumer relies entirely on SNS's own retry schedule
+No consumer-side polling, batching, or backpressure control
+A client-side failure (e.g. deleted endpoint) is not retried at all

SNS + SQS per consumer

−Each queue durably holds messages until its consumer polls them
−Consumer controls its own poll rate, batch size, and retry/backoff
−One consumer falling behind or crashing does not affect the others

Remember: SNS fans one publish out to N subscribers; putting an SQS queue in front of each consumer turns "push and hope" into "push into a durable, pollable buffer per consumer." One queue per consumer for true fan-out — a shared queue splits messages instead of duplicating them. The queue's access policy must allow the SNS service principal to send to it.

standardintermediate

A direct synchronous call means Service A calls Service B and waits for a response — A cannot finish its own work until B answers, and A must know B's address. Publishing to an SNS topic means A hands off a message and moves on immediately — A never learns how many services end up handling it, or how long they take. Pub/sub fits when A does not need an answer to continue, and when more than one downstream service should react to the same event.

Think of it as

A direct call is a phone call: the caller is blocked until the other side picks up and responds, and the caller has to already know the number. A publish to a topic is more like posting a notice on a board: the poster walks away immediately, and any number of people can read the notice later, at their own pace, without the poster ever knowing who.

sync-vs-publish.pypython

# Direct synchronous call — order_service blocks until payment_service responds
response = payment_client.charge(order_id, amount)   # order_service waits here

# Pub/sub — order_service hands off and continues immediately
sns.publish(TopicArn=ORDER_PLACED_TOPIC, Message=json.dumps(order))
# fulfillment and analytics each subscribe independently; order_service
# never learns how many subscribers exist or when they finish

Remember: Choose a direct synchronous call when the caller needs the result to proceed (e.g. an authorization decision). Choose pub/sub (SNS) when the caller does not need a response, or when more than one downstream service should react to the same event without the publisher knowing who they are.