beginnerintermediateadvanced

System Design · Section 47

Rate Limiting

Level: intermediate
Read: 20 min
Concepts: 3

A rate limiter decides whether to allow or reject each request based on recent traffic from the same caller. Four algorithms — fixed window, sliding window, token bucket, leaky bucket — differ in how they track that history and handle bursts, with fixed window's boundary-burst gap being the classic reason to reach for one of the other three. Enforcing a limit correctly across more than one application instance requires either shared state (typically Redis) or a single edge-level chokepoint — an in-process counter silently multiplies the effective limit by the instance count. Finally, what a limit is counted per — user, IP, API key, endpoint, or a combination — determines which abuse scenario it actually defends against.

This section

Rate Limiting system design | TOKEN BUCKET, Leaky Bucket, Sliding LogsTech Dummies - Narendra Lakshmana Gowda

What is true here

Fixed window is simplest but can let through nearly double the limit across a window boundary; sliding window, token bucket and leaky bucket each close that gap differently.
An in-process counter is correct only with exactly one instance — scaling out silently multiplies the effective limit unless enforcement moves to shared state or an edge chokepoint.
A limit is a (key, count, window) triple — the key decides which abuse scenario it actually catches.
Real systems often combine scopes (e.g. per-IP and per-account on a login endpoint) because a single scope alone can be evaded by spreading the attack across the other dimension.

What you will be able to do

Choose the right rate-limiting algorithm for a given burst-tolerance requirement
Explain why an in-process rate limiter breaks silently once an application scales to multiple instances
Pick a limit scope (or combination of scopes) matched to a specific abuse scenario

Algorithms and scaling the enforcement point

The four standard algorithms for deciding allow-or-reject, and why enforcing a limit correctly needs more than one instance's own memory.

Fixed window, sliding window, token bucket and leaky bucket

coreintermediate

A rate limiter decides, for each incoming request, whether to allow it or reject it based on how many requests the same caller has already made recently. The four standard algorithms differ in how they track "recently" and how they handle bursts: fixed window counts requests in a clock-aligned bucket and resets abruptly, sliding window smooths that count across a moving time range, token bucket allows controlled bursts by spending saved-up tokens, and leaky bucket forces a constant output rate regardless of how bursty the input is.

Think of it as

Think of four different door policies for a club with a "50 people per hour" rule. Fixed window is a bouncer who resets his tally sheet to zero the moment the clock hits the new hour — anyone waiting just outside can rush in right at the reset, doubling up on the boundary. Sliding window is a bouncer who always looks back exactly 60 minutes from right now, so there is no reset moment to exploit. Token bucket is a bouncer handing out a fixed number of tickets per minute that pile up if unused, so a quiet hour lets a sudden crowd burst in all at once using saved tickets. Leaky bucket is a single-file turnstile — people can arrive in any clump, but they are let through the door at one constant, unhurried pace no matter how the crowd bunches up outside.

text

fixed window:    count++ in bucket(now / window_size); reset at boundary
sliding window:   count requests where timestamp > (now - window_size)
token bucket:      tokens = min(cap, tokens + elapsed * refill_rate); allow if tokens >= 1
leaky bucket:      queue.push(request); drain queue at fixed_rate; drop if queue full

What we're doing: Show the classic fixed-window boundary-burst problem concretely: a limit of 100 requests/minute lets through nearly 200 in a short span straddling the window edge.

fixed-window-burst.txttext

Limit: 100 requests per minute, fixed window
Window A: 12:00:00 - 12:00:59
Window B: 12:01:00 - 12:01:59

12:00:30 - 12:00:59  -> caller sends 100 requests (all allowed,
                         fills window A's counter to exactly 100)
12:01:00 - 12:01:29  -> caller sends 100 more requests (all
                         allowed, window B's counter starts at 0)

Result: 200 requests allowed in the 60-second span from
12:00:30 to 12:01:29, against a "100 per minute" limit —
because the two 100-request bursts land in two different
counters that reset independently at the clock boundary.

5: All 100 requests in the last 30 seconds of window A are allowed — window A's counter is exactly at the limit, not over it.
7: The moment the clock ticks into window B, the counter resets to 0, so the next 100 requests are allowed too, even though they arrive seconds after the first 100.
10: The 60-second sliding span 12:00:30-12:01:29 saw 200 requests — double the stated limit — purely because it straddles the fixed-window reset point.

Why this works: This is the concrete reason production systems reach for sliding window, token bucket or leaky bucket instead of fixed window when a hard burst cap actually matters — fixed window's simplicity trades away a real guarantee at the boundary, not just a theoretical edge case.

Assuming a fixed-window limiter caps the true worst-case burst at the stated limit

Wrong

text

"We rate-limit to 100 requests/minute with a
fixed window, so no caller can ever exceed 100
requests in any 60-second span."

Better

text

"We rate-limit to 100 requests/minute with a
fixed window — the true worst case is just under
200 requests in a 60-second span straddling a
window boundary. If a hard cap on any rolling
60-second span matters (e.g. protecting a
downstream system's real capacity), use a sliding
window or token bucket instead."

What you see: A downstream system sized for "100 requests/minute" gets overwhelmed by traffic that a fixed-window limiter reported as fully compliant — the spike lands across a window boundary and each half looks fine to its own counter.

Why: A fixed window only bounds the count within each clock-aligned interval independently; it makes no promise about any other 60-second span, including ones that straddle two windows — that gap is exactly where up to double the stated limit can get through.

Four rate-limiting algorithms

Fixed window

resets abruptly at the boundary

Sliding window

a continuously moving range

Token bucket

allows a burst up to bucket size

Leaky bucket

constant, fixed drain rate

The four algorithms compared

The four algorithms compared
Algorithm	Tracks	Handles bursts	Main weakness
Fixed window	Count per clock-aligned interval	Resets abruptly	Boundary burst — up to 2x the limit at window edges
Sliding window	Count over a continuously moving interval	Smooth, no reset spike	More memory/compute than fixed window
Token bucket	Tokens refilled at a fixed rate, up to a cap	Allows a burst up to bucket size	A long idle period lets a large burst through all at once
Leaky bucket	Requests queued, drained at a constant rate	Smooths bursts into steady output	Adds queuing latency; excess requests are dropped, not just delayed, once the queue is full

Remember: Fixed window: simple, but up to ~2x the limit can leak through a window boundary. Sliding window: closes that gap, costs more state. Token bucket: allows bursts up to bucket size after idle time. Leaky bucket: forces a constant output rate, queuing or dropping the rest.

Distributed rate limiting needs shared state or edge-level controls

coreintermediate

A rate limit is only real if it is enforced against the caller's total traffic across every instance handling requests, not against what one instance happened to see. An in-memory counter that lives inside a single application process only ever sees the slice of traffic a load balancer routed to that one instance — with N instances behind a load balancer, the same caller can get roughly N times the intended limit by having requests spread across them. Fixing this means either giving every instance access to one shared counter (typically Redis) or moving enforcement to a single chokepoint all traffic passes through before it ever reaches an instance (an API gateway or edge/CDN layer).

Think of it as

It is like five separate ticket booths for the same concert, each with its own paper tally capped at 100 tickets, and no booth able to see what the others have sold. A determined buyer can walk to each booth and get 100 tickets from every one of them — 500 total — because "the limit" only ever existed independently, per booth. The fix is either one shared ledger all five booths write to before selling a ticket (shared state), or a single entrance gate before any booth that only lets 100 people through in total, so it no longer matters how many booths exist behind it (edge-level enforcement).

text

# in-process (broken once there is more than one instance)
counter = 0  # lives in this process's memory only
if counter >= limit: reject()
counter += 1

# shared-state (correct across instances) — see the Redis
# INCR + EXPIRE pattern already covered under Redis for
# distributed caching, rather than re-deriving it here

What we're doing: Show why an in-process counter silently multiplies the effective limit once an application scales to multiple instances.

in-process-limiter-fails.txttext

Intended limit: 100 requests/minute per API key
Deployment: 3 app instances behind a round-robin load balancer,
            each with its own in-memory counter

Caller sends 300 requests in one minute for the same API key.
Load balancer spreads them ~evenly:
  Instance 1 sees 100 requests -> its local counter allows all 100
  Instance 2 sees 100 requests -> its local counter allows all 100
  Instance 3 sees 100 requests -> its local counter allows all 100

Total allowed: 300 requests/minute for one API key —
3x the intended 100/minute limit, and every individual
instance's logs show it correctly enforcing "100 per minute."

7: Each instance keeps its own counter, so each one independently sees only its third of the traffic — well under its local view of the limit.
11: The true total is 300, three times the intended limit, even though every instance individually behaved exactly as its local rate-limiting code was written to.

Why this works: This is the core failure mode distributed rate limiting exists to fix: the limiter code was not wrong in isolation, but the architecture around it — one counter per instance, no shared visibility — made the enforced limit scale with instance count instead of staying fixed.

Adding more app instances without revisiting an existing in-process rate limiter

Wrong

text

"Our rate limiter has always worked fine, we
just scaled from 1 instance to 6 for the traffic
increase — no changes needed to the limiter."

Better

text

"We're scaling from 1 instance to 6. Our
current rate limiter keeps its counter in process
memory, so the effective per-caller limit is
about to become 6x what we intend. Move the
counter to shared state (Redis) or enforce the
limit at the load balancer/gateway before scaling
out further."

What you see: Abuse or cost-control incidents that "shouldn't be possible given the configured limit" start appearing right around the same time the service was scaled to more instances — the limiter configuration never changed, but the deployment topology did.

Why: An in-process counter's correctness is quietly coupled to instance count — it is entirely correct at one instance and increasingly wrong at every instance added after that, which makes this a bug that surfaces only after an unrelated scaling change, not at the time the limiter was first written.

3 instances, 3 counters — 300 allowed, not 100

In-process counters

Instance 1 (100)

Instance 2 (100)

Instance 3 (100)

In-process counters vs shared state vs edge-level enforcement

In-process counters vs shared state vs edge-level enforcement
Approach	Where the count lives	Cross-instance accurate	Added cost
In-process counter	Memory of one app instance	No — one counter per instance	None, but limit is not real once scaled out
Shared state (e.g. Redis)	One external store, read/written by every instance	Yes	A network round trip per request; the store becomes a dependency
Edge-level (gateway/CDN)	A single chokepoint in front of all instances	Yes, by construction	Coarser control over per-application logic; depends on the edge product's own limits

Remember: An in-process counter is only correct with exactly one instance — scale to N instances and it silently allows up to N times the intended limit. Fix it with shared state (e.g. Redis) that every instance reads and writes, or by enforcing the limit at a single edge/gateway chokepoint before requests ever reach an instance.

Choosing what a limit is counted per

Matching the limit's key to the specific abuse scenario it needs to catch.

Choosing the right limit scope: user, IP, API key, endpoint

standardintermediate

A rate limit is defined by two things: a number, and what that number is counted per — the "key." The same numeric limit means something very different depending on whether it is counted per user account, per IP address, per API key, or per endpoint (or some combination of these). Picking the wrong key either fails to stop the abuse it was meant to prevent, or throttles legitimate traffic that happens to share the key with someone else.

Think of it as

It is like deciding what a speed limit sign actually applies to on a shared road. "Per vehicle" (user/API key) stops one car from speeding no matter how it merges into traffic. "Per lane" (IP address) stops a lane from having too much traffic in aggregate, but if many cars share a lane through a tollbooth (many users behind one corporate NAT), they collectively look like a single fast-moving vehicle and get throttled together, or one bad driver in the lane gets everyone in it flagged. "Per road segment" (endpoint) sets a different limit for the fragile bridge than for the wide highway, because the two segments can tolerate very different amounts of traffic regardless of who is driving.

text

# a limit is a (key, count, window) triple — the key decides the scope
key = user_id                     # per-user
key = client_ip                   # per-IP
key = api_key                     # per-API-key
key = f"{api_key}:{endpoint}"     # per-key, per-endpoint (combined scope)
key = f"{client_ip}:{username}"   # per-IP AND per-account (login example)

What we're doing: Show why a login endpoint needs a combined scope, not a single key, to catch both attack shapes credential-stuffing and brute force represent.

Scenario: attacker attempts credential stuffing —
trying one leaked username/password pair against
many accounts, from a botnet of 500 different IPs,
one attempt per IP per account.

Limit: per-IP only, 5 attempts/minute per IP
  -> each IP only ever makes 1 attempt: NEVER
     triggers the per-IP limit, no matter how many
     accounts are targeted in total.

Limit: per-account only, 5 attempts/minute per
       username
  -> 500 IPs each try a DIFFERENT username once:
     no single username sees more than 1 attempt,
     so this also never triggers.

Limit: per-IP AND per-account together
  -> a single IP hammering ONE account is caught
     by the per-account limit; a single IP trying
     MANY accounts is caught once it exceeds the
     per-IP request volume even spread across
     accounts — the combination closes both gaps
     that either scope alone misses.

8: A pure per-IP limit fails here because the attack is deliberately spread thin across many IPs — each individual IP stays far under any reasonable per-IP threshold.
13: A pure per-account limit fails for the same reason in the other dimension — each individual account only ever sees one attempt.
19: Only the combined key catches both shapes of the attack, because at least one of the two dimensions is forced to accumulate regardless of how the attacker distributes requests.

Why this works: A single-scope limit only bounds abuse along the one dimension it counts — an attacker who can vary the other dimension (many IPs, or many accounts) can stay under any single-scope threshold indefinitely; this is the concrete case for combining scopes rather than picking the "best" single one.

Rate-limiting a login or password-reset endpoint by IP alone

Wrong

text

"We limit /login to 10 attempts/minute per
IP address — that should stop brute-force
attacks."

Better

text

"We limit /login to 10 attempts/minute per IP
(catches a single-source brute force) AND to 5
attempts/minute per targeted username (catches
credential stuffing spread across many IPs at one
account) — the two scopes catch different attack
shapes and neither alone is sufficient."

What you see: Account-takeover attempts succeed at scale despite a "working" rate limiter, because the traffic pattern was distributed across enough source IPs that no individual IP ever crossed the configured threshold.

Why: Per-IP alone assumes the attacker is limited to few IPs, which is false for anyone with access to a botnet, proxy pool, or residential IP rotation service — the missing per-account dimension is exactly what lets a widely distributed attack through undetected.

Four limit scopes, four abuse shapes

Per-user

needs auth already established

Per-IP

weak against shared NAT / rotation

Per-API-key

ties to one integration

Per-endpoint

protects one expensive route

Limit scope matched to the abuse scenario it actually defends against

Limit scope matched to the abuse scenario it actually defends against
Scope	Defends well against	Weak point
Per-user / per-account	One authenticated user overusing the API from any device	Requires authentication to already exist; useless for pre-login abuse
Per-IP	A single-source flood or scraper hitting unauthenticated endpoints	Shared NAT punishes many legitimate users together; easily evaded by rotating IPs
Per-API-key	Runaway or misbehaving third-party integrations; enforcing tiered plans	A leaked key still gets the legitimate holder's full limit until it is revoked
Per-endpoint	Protecting an expensive operation (search, export, password-reset email) independent of caller identity	Does not by itself stop one caller from abusing many different endpoints

Remember: A limit is a (key, count, window) triple — the key is the scope. Per-user/API-key for precise, identity-based control; per-IP for pre-authentication traffic (with shared-NAT and rotation caveats); per-endpoint to protect specific expensive operations regardless of caller. Combine scopes (e.g. IP + account on login) when a single scope leaves a gap an attacker can spread across.