beginnerintermediateadvanced

System Design · Section 52

Backpressure

Level: intermediate
Read: 16 min
Concepts: 3

Backpressure is any mechanism that stops a fast producer from overwhelming a slow consumer — without one, the gap between write rate and drain rate is absorbed by an unbounded queue that grows until the process runs out of memory and crashes. Five concrete mechanisms apply the pressure in practice: a bounded queue gives the mismatch a hard ceiling, a rate limit caps the producer directly, flow control lets the consumer pull only as much as it can handle, batch sizing controls the unit of work moving through the pipeline, and consumer scaling raises drain capacity — real systems combine several rather than relying on one. Monitoring it well means watching queue age (how long the oldest item has waited) and consumer lag (how far behind the consumer is), not just raw queue length, since a short queue can still hide a consumer that has completely stalled.

This section

Understanding BackPressure - Intermediate System Design Episode 2Akhil Sharma

What is true here

A producer and consumer rarely process at the same rate — without a bound, the gap becomes unbounded memory growth, not a self-correcting slowdown.
Bounded queues, rate limits, flow control, batch sizing and consumer scaling apply pressure at different points in the pipeline and are normally combined.
Consumer scaling has a provisioning lag; it complements a bounded queue or rate limit rather than replacing the need for one.
Queue length hides whether a backlog is a problem — queue age and consumer lag expose a stalled consumer that length alone cannot distinguish from a healthy one.

What you will be able to do

Explain why an unbounded queue between a fast producer and a slow consumer eventually causes an out-of-memory crash
Choose and combine bounded queues, rate limits, flow control, batch sizing and consumer scaling for a given pipeline
Avoid treating consumer scaling as a substitute for a hard ceiling on queue growth
Monitor queue age and consumer lag instead of relying on raw queue length to catch a stalled consumer

The mismatch and how to fix it

What backpressure is, the unbounded-growth failure mode without it, and the five mechanisms that apply it in practice.

What backpressure is and why it matters

coreintermediate

Backpressure is any mechanism that stops a fast producer from writing work faster than a consumer can process it. Without one, the gap between the two rates has to live somewhere — usually an in-memory queue that grows until the process runs out of memory and crashes.

Think of it as

Picture a kitchen where the sink drains at a fixed rate and a tap fills it. If the tap runs faster than the drain, the water does not vanish — it rises. Backpressure is turning the tap down, or shutting it off, before the sink overflows, instead of hoping the drain will magically speed up on its own.

text

# Unbounded: producer never checks consumer capacity
while True:
    item = generate_work()
    queue.put(item)          # always succeeds, queue grows forever

# With backpressure: producer is forced to slow down
while True:
    item = generate_work()
    queue.put(item, block=True, timeout=None)  # blocks once queue is full

What we're doing: Show an unbounded in-memory queue growing without limit under a sustained producer/consumer rate mismatch.

unbounded-queue.txttext

queue = []  # plain in-memory list, no size limit

def producer():
    while True:
        queue.append(build_event())   # 1000 events/sec

def consumer():
    while True:
        event = queue.pop(0)
        write_to_database(event)      # 200 events/sec, DB-bound

# producer runs 5x faster than consumer drains --
# queue.append() never fails, so nothing ever tells
# the producer to slow down

1: A plain list has no capacity limit — every append() succeeds regardless of how far behind the consumer already is.
8: The consumer's rate is bound by an external dependency (the database), not by anything the producer can see.
11: This is the core problem: nothing in this code path ever signals "slow down" back to the producer, so the gap only widens.

Why this works: Every message the producer writes while the consumer is behind sits in `queue` in memory. At 800 events/sec of net growth, an hour of sustained mismatch queues 2.88 million events with nothing bounding how large that list can get.

Assuming a queue that "hasn't crashed yet" means the system is keeping up

Wrong

text

queue = []

def producer():
    while True:
        queue.append(build_event())

def consumer():
    while True:
        if queue:
            write_to_database(queue.pop(0))
        # no monitoring on len(queue) at all

Better

text

queue = collections.deque(maxlen=10_000)

def producer():
    while True:
        if len(queue) >= queue.maxlen:
            apply_backpressure()   # block, drop, or reject upstream
        queue.append(build_event())

def consumer():
    while True:
        if queue:
            write_to_database(queue.popleft())
        report_metric('queue.depth', len(queue))

What you see: The service runs fine for hours or days, then the process is suddenly killed by an out-of-memory error with no warning in the logs beyond memory usage that, in hindsight, had been climbing the entire time — because nothing was bounding the queue or alerting on its depth.

Why: An in-memory queue with no capacity limit can absorb an arbitrarily large backlog right up until the process runs out of memory — there is no natural ceiling that forces a decision earlier, so the first sign of trouble is the crash itself, not a graceful degradation.

Same traffic spike, with and without backpressure

No backpressure

+Producer writes at full rate no matter what
+Queue grows without a ceiling during the spike
+Memory usage climbs until the process is killed
+Messages already buffered are lost on crash

With backpressure

−Producer is signaled to slow down or blocked
−Queue depth stays within a known bound
−Memory usage stays flat and predictable
−Producer, not the queue, absorbs the slowdown

The producer/consumer mismatch and its consequence without backpressure

The producer/consumer mismatch and its consequence without backpressure
Scenario	Producer rate vs consumer rate	What happens with no backpressure
Traffic spike	Producer rate jumps well above consumer's steady-state drain rate	Queue depth grows for the duration of the spike, unbounded
Slow downstream dependency	Consumer rate drops (e.g. a database write now takes 5x longer)	Queue depth grows even though producer rate never changed
Consumer crash or restart	Consumer rate drops to zero for a period	Queue depth grows at the full producer rate until the consumer returns

Remember: Backpressure exists because a producer and consumer rarely process at the same rate, and the gap has to go somewhere. Without a bound on the queue, "somewhere" is process memory, and the failure looks like a sudden crash after a period of no visible warning.

Backpressure mechanisms: bounded queues, rate limits, flow control, batch size, consumer scaling

coreintermediate

Five mechanisms apply backpressure in practice: a bounded queue gives the mismatch a hard ceiling, a rate limit caps the producer directly, flow control lets the consumer pull only as much as it can handle, batch sizing controls how much work one unit of processing represents, and consumer scaling adds drain capacity. Real systems combine several rather than relying on just one.

Think of it as

Think of a highway on-ramp with a metering light. A bounded queue is the ramp itself — only so many cars can wait there before it backs onto the street. A rate limit is the metering light's fixed interval. Flow control is a smarter light that reads live highway speed and adjusts the interval itself. Batch size is how many cars are waved through per green light. Consumer scaling is opening another highway lane. None of them work by pretending traffic will simply thin out on its own.

text

# Flow control: consumer requests exactly what it can handle
subscriber.request(100)          # "send me up to 100 items"
producer.onRequest(n -> {
    send(next(n));                # never sends more than requested
});

# Rate limit: producer capped independent of queue state
if not rate_limiter.allow():
    reject_or_delay(item)
else:
    queue.put(item)

What we're doing: Combine a bounded queue, a batch size, and consumer scaling to keep a log-ingestion pipeline stable under a traffic spike.

combined-mechanisms.txttext

queue = BoundedQueue(max_size=20_000)   # 1: hard ceiling
BATCH_SIZE = 200                        # 3: unit of work per consumer pass

def producer(event):
    if not rate_limiter.allow():        # 2: caps producer rate directly
        return reject(event)
    queue.put(event, block=True, timeout=2)

def consumer_worker():                  # 4: one of N parallel workers
    while True:
        batch = queue.get_batch(BATCH_SIZE)
        write_batch_to_database(batch)  # one round trip per 200 events

# 5: scale N workers based on queue depth, not a fixed number
for _ in range(autoscale_target(queue.depth())):
    spawn(consumer_worker)

1: The bounded queue is the ceiling every other mechanism works within — nothing here can grow memory usage past max_size regardless of what upstream does.
5: The rate limiter acts before anything reaches the queue, so a sustained spike is capped at the source rather than absorbed and dealt with later.
10: Batching 200 events per database write cuts round trips by 200x versus one write per event, at the cost of up to one batch's worth of latency.
16: Worker count is driven by current queue depth, not a static number chosen once — this is consumer scaling responding to the actual mismatch in real time.

Why this works: No single mechanism handles every failure mode alone: the rate limit protects against a sustained spike, the bounded queue protects against any spike the rate limit still lets through, batching cuts steady-state cost, and scaling workers adapts drain capacity to current load — together they bound memory, bound latency, and keep throughput high.

Increasing batch size to fix slow throughput without checking consumer memory or latency budget

Wrong

text

BATCH_SIZE = 200

# throughput looks low, so just make batches bigger
BATCH_SIZE = 50_000  # no check against consumer's
                      # memory limit or downstream
                      # per-request timeout

Better

text

BATCH_SIZE = 200

# raise incrementally, measuring consumer memory
# and end-to-end latency at each step
for candidate in [500, 1_000, 2_000]:
    measure_latency_and_memory(candidate)
BATCH_SIZE = largest_candidate_within_budget

What you see: Throughput improves briefly, then consumer workers start failing with out-of-memory errors or the downstream database rejects oversized batch writes with a timeout — because a batch fifty times larger than before also multiplies the memory and per-request time needed to hold and process it.

Why: Batch size is a trade-off, not a free lever: a larger batch amortizes fixed overhead (connection setup, per-call latency) across more items, but every item in the batch still occupies memory simultaneously and the whole batch fails or succeeds together, so growing it without a bound just moves the same unbounded-growth problem from the queue to the batch.

Where each mechanism sits in the pipeline

Producer

rate limit applied here

Bounded queue

batch size groups items here

Flow control

consumer signals demand

Consumers

scaled out for drain rate

The five mechanisms and where each one applies pressure

The five mechanisms and where each one applies pressure
Mechanism	Where it acts	Trade-off
Bounded queue	Caps how much unprocessed work can accumulate	Full queue forces an explicit choice: reject, block, or drop
Rate limit	Caps the producer's write rate directly	Simple and predictable, but a fixed cap can throttle legitimate spikes
Flow control	Consumer pulls only what it can handle (demand-based)	More precise than a fixed rate limit, but needs a protocol both sides support
Batch size	Controls how much work one processing unit represents	Bigger batches cut overhead but raise per-batch latency and memory
Consumer scaling	Raises total drain rate by adding workers or partitions	Only helps if the work can be safely parallelized

Remember: A bounded queue, a rate limit, flow control, batch sizing, and consumer scaling each apply pressure at a different point in the pipeline. Real systems combine several — a bounded queue as the hard ceiling, plus a rate limit or flow control to keep the queue from filling in the first place, plus scaling to raise drain rate over time.

Watching it actually work

Why raw queue length is a weaker signal than queue age and consumer lag.

Monitoring backpressure: queue age and lag, not only queue length

standardintermediate

Queue length alone hides whether a backlog is actually a problem — the more honest signals are queue age (how long the oldest unprocessed item has been waiting) and consumer lag (how far behind the consumer is, in messages or time). A short queue can still mean an unacceptable wait if the consumer is stuck.

Think of it as

A short line at a coffee shop looks fine from the length alone — but if the barista has been stuck on one order for twenty minutes, the person at the front has been waiting twenty minutes no matter how short the line looks. Counting people in line tells you volume; timing how long the first person has waited tells you whether service is actually working.

text

# Queue length -- cheap, but incomplete
metric('queue.length', queue.size())

# Queue age -- the oldest item's wait time
oldest = queue.peek_oldest()
metric('queue.oldest_age_seconds', now() - oldest.enqueued_at)

# Consumer lag (Kafka-style) -- offset gap per partition
lag = latest_offset(partition) - committed_offset(consumer_group, partition)
metric('consumer.lag', lag)

What we're doing: Read Kafka consumer lag per partition to detect a consumer that has stopped making progress.

consumer-lag-check.txttext

# kafka-consumer-groups.sh --describe --group order-processor
GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-processor  orders  0          998042           998050          8
order-processor  orders  1          871200           1050300         179100
order-processor  orders  2          502991           502995          4

# Partition 1's lag of 179,100 messages, growing on
# every check, means that consumer instance has
# effectively stopped -- even though partitions 0
# and 2 look completely healthy

4: CURRENT-OFFSET is what the consumer group has committed; LOG-END-OFFSET is the latest message actually written — LAG is the gap between them.
5: A lag of 179,100 on one partition while siblings show single digits means the problem is localized to whichever consumer instance owns partition 1, not the topic as a whole.

Why this works: Per-partition lag isolates the problem to the exact consumer instance that stalled — a topic-wide "queue length" metric would have blended partition 1's stall in with the two healthy partitions and likely stayed inside a normal-looking range.

Same queue length, opposite reality

Queue length: 500 (both cases)

+Case A: draining at 1,000/sec — clears in 0.5s
+Case B: consumer stuck, draining at 0/sec — growing stale
+Length alone reports "500" for both, no distinction

Oldest-item age tells them apart

−Case A: oldest item is 0.4s old — healthy
−Case B: oldest item is 20 minutes old — stuck consumer
−Age (or lag) exposes the real state length hides

The three signals and what each one actually tells you

The three signals and what each one actually tells you
Signal	What it measures	What it misses
Queue length	Raw count of unprocessed items right now	Says nothing about drain rate or how long items have waited
Queue age	How long the oldest unprocessed item has been waiting	Needs a per-item timestamp; not always tracked by default
Consumer lag	How far behind the consumer is vs. the producer (messages or time)	Requires the broker/queue to expose committed offset vs. latest offset

Remember: Queue length tells you volume, not health. Queue age (oldest item's wait time) and consumer lag (offset or time behind the producer) tell you whether the consumer is actually keeping up — monitor those, and alert on them, not just the raw count.