beginnerintermediateadvanced

System Design · Section 77

Load Shedding

Level: intermediate
Read: 12 min
Concepts: 2

Load shedding starts from a counter-intuitive but repeatedly observed result: under severe overload, a system that tries to accept and process everything arriving at it often ends up serving close to zero requests successfully, because queueing delay, resource contention and retry storms compound rather than simply degrading proportionally — while a system that deliberately rejects the excess, quickly and cheaply, keeps the accepted fraction running at full, healthy latency and frequently achieves far higher total successful throughput as a result. Making that rejection decision well, rather than randomly, is what the second half of this section covers: priorities rank incoming work so shedding targets the least valuable requests first rather than treating a checkout and a background analytics ping as equally disposable; quotas cap how much of the system's capacity any single caller can consume, preventing one runaway or abusive client from looking like a genuine system-wide capacity shortfall; admission control is the actual enforcement point — the place a request is accepted or rejected, before it consumes any processing resources, based on current capacity plus the request's priority and the caller's quota standing; and bounded queues give admission control a cheap, immediate signal (is the queue full right now) rather than relying on slower, more expensive signals like rising average latency that only reveal overload well after it has already begun. The four mechanisms compose at one enforcement point rather than functioning as independent, alternative choices.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Under severe overload, accepting everything can collapse successful throughput toward zero; deliberately shedding the excess keeps the accepted fraction healthy and often raises total successful throughput.
A shed request must be genuinely cheap to reject — an expensive rejection defeats the purpose of shedding at all.
Priorities let shedding target the least valuable work first, rather than rejecting valuable and worthless requests at random.
Quotas cap per-caller consumption so one caller cannot monopolize capacity or masquerade as a system-wide capacity crisis.
Bounded queues give admission control an immediate, cheap overload signal — queue depth — rather than requiring overload to be inferred from slower signals like latency.

What you will be able to do

Explain why accepting all incoming work under severe overload can produce lower total successful throughput than deliberately shedding some of it
Design a load-shedding gate whose rejection path is genuinely cheap, not a hidden new source of contention
Combine priorities, quotas, admission control and bounded queues into a single, fair shedding policy
Calibrate a per-caller quota tight enough to actually protect shared capacity

Why shed on purpose

The counter-intuitive result: rejecting excess work deliberately usually beats trying to serve everyone.

Load shedding: rejecting work on purpose

coreintermediate

Load shedding is the deliberate decision to reject some incoming work when a system does not have the capacity to handle everything arriving at it, rather than trying to accept and process everything and letting quality degrade uniformly (or collapse entirely) under the overload. The counter-intuitive part is that rejecting a controlled fraction of requests can produce a far better overall outcome than accepting all of them: a system that tries to serve 100% of requests at 300% of its capacity typically ends up serving close to 0% successfully, because queues grow unboundedly, latency climbs until clients time out anyway, and resource contention (thread pools, connections, memory) grinds every request to a halt including the ones that would have succeeded fine on their own. A system that instead sheds the excess — say, rejecting the top 60% of requests immediately with a fast, cheap "try again later" response — keeps the remaining 40% running at full, healthy speed. The total number of successfully-served requests is often much higher with shedding than without it, because the alternative to "reject 60% cleanly" is not "serve 100% slowly," it is "serve close to 0% because the system fell over."

Think of it as

A lifeboat rated for 50 people that tries to take on 80 sinks and saves nobody; a lifeboat that turns away the last 30 and stays at its rated capacity saves 50. Load shedding is the crew member at the boarding point making the hard, deliberate call to turn people away once the boat is full, rather than letting everyone climb aboard and watching the boat go under with everyone still on it. The number who get turned away (30) is a real, visible cost — but it is a far smaller cost than the alternative, and it is a cost paid on purpose, by someone making a decision, rather than an accident that happens to everyone including the 50 who would otherwise have been fine.

python

# A minimal load-shedding gate at the edge of a
# request handler: reject immediately once a
# cheap load signal crosses a threshold
def handle_request(request):
    if current_queue_depth() > MAX_HEALTHY_DEPTH:
        return Response(status=503,
                         body="capacity exceeded, retry later",
                         retry_after_seconds=5)
    return process(request)  # only reached when
                              # capacity genuinely allows it

What we're doing: Compare total successful throughput for a service rated at 1,000 req/s receiving 3,000 req/s, with and without shedding.

overload-throughput-comparison.txttext

Service capacity: 1,000 req/s (healthy latency at
or below this rate)
Incoming traffic: 3,000 req/s (3x overload)

Without shedding:
  All 3,000 req/s accepted -> queue grows unbounded ->
  average latency climbs past client timeout (2s) ->
  clients time out and many retry, adding MORE load ->
  measured successful throughput after 60s: ~80 req/s

With shedding (reject at 1,000 req/s, fast 503s
for the rest):
  1,000 req/s processed normally at healthy latency ->
  2,000 req/s rejected immediately with a cheap 503 ->
  measured successful throughput: ~1,000 req/s

9: This is the collapse this concept opened with: the unshed system does not serve a degraded-but-reasonable fraction of the 3,000 req/s, it serves almost none of it, because the queue and retry storm consume capacity that would otherwise have gone toward completing requests.
14: The shedding system serves roughly its full rated capacity, because the rejected 2,000 req/s never entered the queue or consumed processing resources at all — the fast, cheap rejection is what keeps the accepted work isolated from the excess.

Why this works: The comparison is not "1,000 served vs. 3,000 served at lower quality" — it is "1,000 served vs. ~80 served," which is the actual, counter-intuitive result overload testing repeatedly finds: past a certain point, trying to serve more work than a system can handle serves less total work than deliberately refusing the excess.

Making a rejected request expensive to reject

Wrong

python

def handle_request(request):
    if current_queue_depth() > MAX_HEALTHY_DEPTH:
        log_rejection_to_database(request)  # a DB
        write, itself competing for the same
        overloaded resources, just to reject
        return Response(status=503)
    return process(request)

Better

python

def handle_request(request):
    if current_queue_depth() > MAX_HEALTHY_DEPTH:
        increment_local_counter('rejected')  # cheap,
        # in-memory, no shared-resource contention
        return Response(status=503,
                         retry_after_seconds=5)
    return process(request)

What you see: A load-shedding mechanism is added, but the system still collapses under overload, because the rejection path itself writes to the same database connection pool that the accepted requests are competing for — the "cheap" rejection turned out to consume real, contended capacity.

Why: The entire value of load shedding depends on the rejection being genuinely cheap — if rejecting a request costs nearly as much as processing one, shedding does not actually free up the capacity it is supposed to protect, and the system can still saturate even while nominally "shedding" load.

Accept everything vs. shed the excess, under the same 3x overload

Accept everything

+Queue depth grows without bound
+Latency climbs until clients time out anyway
+Resource contention slows every request, including ones that would have succeeded alone
+Successful throughput can collapse toward zero

Shed the excess

−Requests beyond capacity are rejected immediately, cheaply
−The accepted fraction runs at full, healthy latency
−No queue growth, no resource contention spiral
−Total successful throughput is often much higher

Accepting everything vs. shedding under 3x overload

Accepting everything vs. shedding under 3x overload
Approach	Requests accepted	Typical outcome under severe overload
Accept everything	100% attempted	Queues and resource contention compound; successful throughput often collapses toward 0%
Shed the excess	~33% attempted (matched to capacity)	The accepted fraction completes at full, healthy latency; the rest fail fast and cheaply

Remember: Under severe overload, accepting everything usually collapses successful throughput toward zero, because queueing, contention and retries compound — deliberately rejecting the excess (fast and cheaply) keeps the accepted fraction running at full health, and often produces far higher total successful throughput than trying to serve everyone. A rejection has to genuinely be cheap to reject, or shedding does not actually relieve the pressure it is meant to.

Making shedding fair and cheap

Priorities, quotas, admission control and bounded queues, composed at one enforcement point.

Priorities, quotas, admission control and bounded queues

coreintermediate

Deciding to shed load (the prior concept) still leaves the question of exactly what to shed and how — four mechanisms answer that together. Priorities rank work so that shedding is not random: a checkout request, a background sync job, and an analytics event are not equally valuable, and a priority label lets the system shed the lowest-priority work first, protecting what actually matters most under overload. Quotas cap how much of a resource a single caller (a user, a tenant, an API key) can consume, which prevents one caller from monopolizing capacity that other callers need — without a quota, one runaway or abusive client can look, from the system's perspective, exactly like a genuine capacity shortfall, when the real problem is one caller taking far more than a fair share. Admission control is the general mechanism that actually rejects a request at the door, before it enters processing at all, based on current capacity, priority, and quota — it is the point where the decision "accept or shed" is actually made. Bounded queues put a hard ceiling on how much work is allowed to wait rather than allowing an unbounded backlog to grow — once the queue is full, admission control has an immediate, cheap signal to shed on (the queue is full) rather than needing to infer overload from slower, more expensive signals like rising latency.

Think of it as

A hospital emergency room under a surge of patients does not treat everyone in arrival order (that would be a plain, unbounded queue) — it triages by severity (priority: the most urgent cases go first, not the earliest arrivals), limits how many non-critical cases any one referring clinic can send in at once during a surge (a quota, so one source cannot flood the ER and starve everyone else), turns away new non-urgent arrivals at the door once the waiting room is genuinely full rather than letting people wait indefinitely in a hallway (admission control against a bounded queue), and caps the physical waiting room itself at a fixed number of chairs so "how full are we" is always answerable at a glance rather than an ever-growing, unbounded crowd (a bounded queue). Every one of these mechanisms is a specific answer to "who gets seen, and who gets turned away, when there is not enough capacity for everyone" — none of them are optional if the ER wants a surge to end in "we treated the sickest patients well" rather than "we tried to see everyone and the whole department ground to a halt."

python

# Admission control composing all four mechanisms
def admit(request):
    if queue.depth() >= MAX_QUEUE_DEPTH:            # bounded queue
        if request.priority != 'critical':
            return reject("queue full")
    if quota.remaining(request.caller_id) <= 0:      # quota
        return reject("quota exceeded")
    if system_overloaded() and request.priority == 'low':  # priority
        return reject("shedding low-priority work")
    queue.enqueue(request)                            # admission
    return accept()                                    # control

What we're doing: Trace three requests through admission control during an overload event: one critical, one from a caller over quota, one low-priority.

admission-control-trace.txttext

System under 2.5x overload, queue at 95% of its
bounded capacity.

Request A: priority=critical, caller within quota
  -> queue check passes (critical bypasses the
     near-full threshold), quota OK, priority OK
  -> ADMITTED

Request B: priority=normal, caller has exceeded its
per-minute quota
  -> quota check fails
  -> REJECTED (quota exceeded)

Request C: priority=low, caller within quota, but
system is actively shedding low-priority work
  -> priority check fails under current load
  -> REJECTED (shedding low-priority work)

5: A critical request is allowed to bypass the near-full queue threshold specifically because priority is one of the signals admission control uses — without it, a genuinely urgent request would be rejected purely because it happened to arrive when the queue was nearly full, with no regard for how important it actually was.
11: This caller is rejected for exceeding its own quota, independent of overall system load — this is what prevents one noisy or misbehaving caller from consuming capacity that should be available to every other caller.

Why this works: All three requests hit the same admission-control function, but each is rejected or admitted for a different one of the four reasons — this is the actual value of combining the mechanisms rather than using just one: a system with only a bounded queue (no priority) would have rejected the critical request in line with everyone else once the queue got full, which is exactly the outcome priority-aware shedding is meant to prevent.

Using an unbounded queue and inferring overload only from rising latency

Wrong

python

def handle_request(request):
    queue.enqueue(request)  # no depth limit at all
    return process_from_queue()
    # overload only becomes visible once average
    # latency has already climbed significantly

Better

python

def handle_request(request):
    if queue.depth() >= MAX_QUEUE_DEPTH:
        return reject("queue full")  # immediate,
        # cheap signal -- no need to wait for
        # latency to reveal the problem
    queue.enqueue(request)
    return process_from_queue()

What you see: A queue grows to hundreds of thousands of pending items during a traffic spike before anyone notices, because the only overload signal being watched was average request latency, which stayed misleadingly normal for the requests already being served while an enormous, invisible backlog built up behind them.

Why: An unbounded queue has no natural signal that anything is wrong until the backlog is already large enough to affect memory, processing time, or eventually latency for everyone — a bounded queue turns "is the system overloaded" into an immediate, cheap, always-current fact (is the queue full), rather than something that has to be inferred indirectly and after the fact.

One request, checked against all four mechanisms before admission

Incoming request

Bounded queue check

is there room to wait at all?

Quota check

has this caller used its fair share?

Priority check

is this worth serving under current load?

Admission control decision

accept or reject

Accepted

Rejected (fast, cheap)

Four mechanisms and the specific question each one answers

Four mechanisms and the specific question each one answers
Mechanism	Question it answers	Failure mode without it
Priorities	Which work matters more, if not everything can be served?	Random shedding rejects valuable and worthless work equally
Quotas	Is one caller consuming an unfair share of capacity?	One abusive/runaway caller looks like a system-wide capacity crisis
Admission control	Accept or reject this request, right now, before processing it?	Overload is only discovered after work has already begun consuming resources
Bounded queues	How much work is allowed to be waiting at once?	An unbounded backlog grows silently until latency or memory collapse reveals it

Remember: Priorities decide what to shed first (least valuable work, not random work); quotas cap what any single caller can consume, so one caller cannot masquerade as a system-wide capacity crisis; admission control is the actual accept-or-reject decision point, made before processing begins; and bounded queues give admission control a cheap, immediate overload signal instead of relying on slower signals like rising latency. All four compose together at one enforcement point.