beginnerintermediateadvanced

System Design · Section 27

Messaging and Queues

Level: intermediate
Read: 22 min
Concepts: 3

Synchronous request/response blocks the caller until the callee replies — simple, but directly coupled to the callee's speed and availability. Asynchronous messaging hands work off through a queue and continues immediately, decoupling the caller from how long the work takes. A queue's vocabulary — producer, consumer, broker, partition, offset, acknowledgement, visibility timeout, retry, dead-letter queue — describes the mechanics of getting a message reliably from send to successful processing. Queues solve three related problems: decoupling producer from consumer, absorbing traffic bursts by draining at a sustainable rate, and moving slow, non-critical work off the request path.

This section

System Design - Part 18 | Kafka | Distributed Messaging Architecture | Queues and StreamsNikhil Lohia

What is true here

Synchronous fits work the caller needs a result from now; asynchronous fits everything else.
Nine terms — producer, consumer, broker, partition, offset, ack, visibility timeout, retry, dead-letter queue — describe reliable message delivery.
A queue decouples producer and consumer, absorbs bursts, and moves slow work off the critical request path.
A queue buys time, not capacity — sustained elevated load still needs the consumer side to scale.

What you will be able to do

Decide whether a given piece of work belongs on the synchronous or asynchronous path
Use the queue vocabulary precisely — offset, visibility timeout, dead-letter queue — when reasoning about delivery
Explain the three distinct problems a queue solves between two services
Recognize when a queue alone won't be enough and consumer capacity also needs to scale

The core choice, and the vocabulary

When to block on a result versus hand work off, and the nine terms that describe reliable delivery mechanics.

Synchronous request/response vs asynchronous messaging

corebeginner

Synchronous request/response means the caller waits for the callee to finish and reply before continuing — simple, but the caller is blocked for as long as the work takes, and a slow or down dependency directly slows or breaks the caller. Asynchronous messaging means the caller hands off a message and continues immediately, with the work happening independently — the caller is decoupled from how long the work takes, at the cost of not having an immediate result.

Think of it as

Synchronous is a phone call — you stay on the line until the other person answers your question, and if they take forever, you're stuck waiting. Asynchronous is sending a text message — you send it and move on with your day, trusting it'll be read and acted on, without knowing exactly when.

text

sync:  client -> service -> [wait] -> response -> client
async: client -> queue -> [continue immediately]
                    ↓
                 worker -> processes later

What we're doing: Show the same feature (sending a welcome email on signup) built synchronously vs asynchronously.

sync-vs-async-signup.txttext

Synchronous signup:
1. Client submits signup form.
2. Server creates the user record.
3. Server calls the email provider's API and WAITS
   for it to confirm the email was sent.
4. Server responds to the client only after step 3.
   If the email provider is slow (2s) or down, every
   signup is now slow or fails — for a step the user
   doesn't need to wait for at all.

Asynchronous signup:
1. Client submits signup form.
2. Server creates the user record.
3. Server publishes a "send welcome email" message
   to a queue, and responds to the client immediately.
4. A separate worker picks up the message and sends
   the email whenever it can — the signup itself is
   never slowed down or broken by the email provider.

4: This is the direct coupling: the signup request's latency now includes a dependency the user never asked to wait on.
15: The queue is what breaks that coupling — signup succeeds independently of the email provider's speed or availability.

Why this works: This is the single most common refactor from synchronous to asynchronous — a non-essential side effect (sending an email) was blocking a critical path (completing signup) for no reason the user would ever notice or want.

Making a non-critical side effect synchronous on the critical request path

Wrong

text

def handle_signup(user_data):
    user = create_user(user_data)
    send_welcome_email(user)  # blocks; a slow
                               # email API slows
                               # every signup
    return success_response(user)

Better

text

def handle_signup(user_data):
    user = create_user(user_data)
    queue.publish("send_welcome_email", user.id)
    return success_response(user)  # returns
                                     # immediately

What you see: A feature that has nothing to do with a user-visible outcome (an email, an analytics event, a search-index update) shows up as the slowest step in a request's trace, and an outage in that unrelated dependency takes down an otherwise-unrelated critical flow.

Why: Not every step in a request handler needs to complete before the response is returned — only steps the caller genuinely needs a result from belong on the synchronous path; everything else is a candidate to move off it via a queue.

Signup: synchronous vs asynchronous email

Synchronous

+Server waits for the email provider to confirm
+A slow or down provider slows or breaks every signup
+Caller is blocked for the full duration

Asynchronous

−Server publishes to a queue and returns immediately
−A worker sends the email independently, whenever it can
−Signup succeeds regardless of the email provider's speed

Synchronous vs asynchronous, at a glance

Synchronous vs asynchronous, at a glance
Property	Synchronous	Asynchronous
Caller waits?	Yes, for the full duration	No, hands off and continues
Coupling to callee's latency	Direct — caller is only as fast as the callee	Decoupled — caller's response time is independent
Failure propagation	A callee failure is immediately visible to the caller	Failure can be retried/handled without the caller ever knowing
Good fit	Work the caller needs a result from right now	Work that can happen after the caller's request completes

Remember: Synchronous: caller waits, simple, but directly coupled to the callee's speed and availability. Asynchronous: caller continues immediately, decoupled, but has no immediate result — use it for work the caller does not need a result from before proceeding.

The queue vocabulary: producer, consumer, broker and the delivery mechanics

coreintermediate

These nine terms describe every part of how a message gets from a producer to being successfully processed. A producer sends messages; a broker stores and routes them; a consumer reads and processes them. The remaining terms describe the mechanics of reliable delivery: how a broker tracks position (partition, offset), how a consumer confirms success (acknowledgement, visibility timeout), and what happens on failure (retry, dead-letter queue).

Think of it as

A broker is a coat-check counter. The producer drops off a coat (message); the counter (broker) holds it and hands the claim ticket's position (offset) to whoever comes to collect. When a consumer takes a coat off the rack, the counter holds it aside for them briefly (visibility timeout) rather than handing it to someone else — but if that consumer never confirms they actually took it (no ack), the counter puts it back on the rack for the next person (retry). A coat nobody can ever successfully claim, after enough attempts, gets moved to a separate lost-and-found shelf (dead-letter queue) instead of blocking the rack forever.

text

producer → broker (partition 0..N, tracked by offset)
                ↓
             consumer receives, gets a visibility timeout
                ↓
        ack (success, remove) | timeout expires → redeliver
                ↓ (after max retries)
           dead-letter queue

What we're doing: Walk through a message's full lifecycle including a failed first attempt.

message-lifecycle.txttext

1. Producer sends "process_order:42" to the queue.
2. Broker stores it, assigns it offset 1057 in
   partition 2.
3. Consumer A receives it — broker starts a 30s
   visibility timeout, hiding it from other consumers.
4. Consumer A crashes mid-processing, never sends an
   ack.
5. 30 seconds pass. The visibility timeout expires —
   broker makes the message visible again.
6. Consumer B receives the same message (this is now
   its 2nd delivery attempt) and processes it
   successfully.
7. Consumer B sends an ack — broker removes the
   message. If step 6 had also failed, and this
   exceeded the configured max-retry count, the
   message would go to the dead-letter queue instead
   of being redelivered a third time.

6: The crash with no ack is exactly what the visibility timeout exists to recover from — no ack means the broker assumes it needs to try again.
12: This is the second delivery attempt — the message was never lost, just held back and retried by a different consumer.

Why this works: This full lifecycle is what "at-least-once delivery" actually looks like mechanically — the visibility timeout and retry are what make a crashed consumer recoverable without losing the message, at the cost of possible duplicate delivery.

Setting a visibility timeout shorter than the actual processing time

Wrong

text

-- visibility timeout: 5 seconds
-- actual processing time: 12 seconds (a slow
-- external API call inside the handler)

Better

text

-- visibility timeout: set comfortably above
-- the 99th-percentile processing time (e.g.
-- 30-60s for a ~12s typical job), or extend it
-- programmatically for jobs that legitimately
-- run long

What you see: The same message gets processed multiple times by different consumers even though nothing actually failed — a slow-but-successful handler is still running when the visibility timeout expires and a second consumer picks up the "abandoned" message.

Why: A visibility timeout shorter than real processing time causes the broker to assume a consumer died and redeliver the message while the original consumer is still legitimately working on it — this is a self-inflicted duplicate-delivery problem, not a real failure.

A message's lifecycle through the queue

Producer

sends the message

Broker

stores at an offset

Consumer

visibility timeout starts

Acked

removed, done

Dead-letter queue

after max retries

The nine terms, grouped by role

The nine terms, grouped by role
Term	Role
Producer	Sends messages
Consumer	Reads and processes messages
Broker	Stores and routes messages between the two
Partition	A subdivision of a queue/topic for parallel processing
Offset	A consumer's read position within a partition
Acknowledgement	Consumer's confirmation that a message was handled
Visibility timeout	How long a message stays hidden after delivery, before becoming re-deliverable
Retry	Reprocessing a failed or un-acked message
Dead-letter queue	Where a message goes after exhausting its retries

Remember: Producer sends, broker stores/routes, consumer reads. Offset tracks position within a partition. Ack confirms success; visibility timeout governs redelivery on silence; retries are bounded, with a dead-letter queue catching what exceeds the limit.

What a queue actually buys a design

The three problems queues solve, and the capacity trade-off they don't remove.

Using queues to decouple services, absorb bursts and move slow work off the request path

coreintermediate

A queue between two components solves three related problems at once: it lets the producer and consumer scale, fail and deploy independently (decoupling), it lets a burst of incoming work be absorbed and processed at a steady rate rather than overwhelming a downstream service (buffering), and it lets slow, non-critical work happen after a request has already returned to the caller (moving work off the critical path).

Think of it as

A queue is like a restaurant's order ticket rail between the front of house and the kitchen. Waiters (producers) hand off orders without needing the kitchen to be ready that instant. If ten tables order at once, the rail absorbs the burst — the kitchen (consumer) works through tickets at its own steady pace instead of being overwhelmed the moment all ten arrive. And the waiter is free to serve other tables immediately, instead of standing at the kitchen window waiting for each dish.

What we're doing: Show a queue absorbing a traffic burst that would otherwise overwhelm a downstream image-processing service.

burst-absorption.txttext

Image-processing service can sustainably handle
50 images/sec (its steady-state throughput).

A marketing campaign causes 2,000 image uploads in
the first 10 seconds — a burst of 200 images/sec,
4x the service's sustainable rate.

Without a queue: uploads are processed synchronously
by the image service directly — it falls over or its
latency spikes massively trying to keep up with 4x
its capacity.

With a queue: all 2,000 uploads are queued
immediately (queuing is cheap and fast). The
image-processing workers drain the queue at their
sustainable 50/sec, finishing the whole burst in
40 seconds — slower per-image completion during
the burst, but the service itself never falls over.

8: This is the failure the queue prevents — a burst hitting a service at a rate it cannot sustain.
14: The trade-off is visible here: individual images take longer to finish during the burst, in exchange for the service staying healthy throughout.

Why this works: A queue turns "the system falls over under a burst" into "processing is temporarily slower during a burst" — a strictly better failure mode for most workloads, since a slow-but-working system usually beats an overloaded, failing one.

Adding a queue but sizing consumer capacity for average load, not peak

Wrong

text

"We queue image processing now, so bursts are
handled." (consumer count sized only for the
50/sec average, never revisited)

Better

text

"We queue image processing, AND autoscale
consumer workers based on queue depth/age, so
a sustained burst (not just a brief spike)
still drains in a reasonable time instead of
the backlog growing indefinitely."

What you see: A queue's backlog keeps growing throughout a sustained traffic increase rather than draining — the queue absorbed the instantaneous burst just fine, but consumer capacity was never scaled to match a burst that turned out to be sustained rather than brief.

Why: A queue buys time to catch up, it does not by itself increase processing capacity — if the elevated load is sustained rather than a brief spike, the consumer side still needs to scale (more workers, autoscaling on queue depth) or the backlog simply grows without bound.

A queue absorbing a burst

Uploads

200/sec burst

Queue

buffers the burst

Workers

drain at 50/sec

Three problems a queue solves, with a concrete example each

Three problems a queue solves, with a concrete example each
Problem	Without a queue	With a queue
Decoupling	Producer fails/slows if the consumer is down	Producer keeps working; messages wait for the consumer to recover
Burst absorption	A traffic spike directly overloads the downstream service	The spike queues up; the consumer drains it at a sustainable rate
Off request path	A slow step (email, PDF generation) blocks the response	The response returns immediately; a worker handles the slow step after

Remember: A queue decouples producer from consumer, absorbs bursts by letting the consumer drain at its own sustainable rate, and moves slow non-critical work off the request path — but it only buys time, so consumer capacity still has to scale to match sustained (not just momentary) load.