beginnerintermediateadvanced

System Design · Section 89

Rate Limiter Design

Level: intermediate
Read: 16 min
Concepts: 2

Designing a rate limiter is six decisions taken in order: the identity key a request is counted against, the scope the limit protects, the algorithm that measures the count over time, the storage the counter lives in, the coordination between enforcement points, and the failure mode. The first five are usually specified; the sixth almost never is, and it exists anyway — whatever the code happens to do when the counter store raises. It has two answers and both are right somewhere: fail open keeps the product working during a store outage and removes protection at exactly the moment a system is least healthy, while fail closed preserves protection and converts a cache outage into a product outage. So the choice is made per limit rather than once: a login-attempt limiter fails closed because it is the only defence against credential stuffing, a generous fairness quota fails open because a fairness rule is not worth an outage. Two implementation facts then decide whether the limiter limits anything. The check is a read-modify-write, and split across separate round trips it lets concurrent requests interleave and overshoot by roughly the in-flight count — a bug proportional to concurrency, so it passes every sequential test and appears at exactly the load the limit exists for. The fix is one indivisible operation: an atomic increment for fixed-window counting, or a server-side script for a token bucket, which needs refill arithmetic against a stored timestamp — with the key's expiry set in the same step, since a crash between INCR and EXPIRE leaves a counter that never resets and blocks a caller permanently. And the enforcement layer decides what the limiter protects: a limit applied inside the application has already spent a TLS handshake, a load-balancer hop, a pooled connection and a worker, so it enforces fairness while capacity stays saturated. Coarse limits at the edge reject before any of that is paid; fine, identity-aware limits stay in the application, where the user, plan and operation cost are known.

This section

Design a Distributed Rate Limiter w/ a Ex-Meta Staff Engineer: System Design BreakdownHello Interview

What is true here

The failure mode is the sixth decision and the one usually left implicit; choose fail-open or fail-closed per limit, based on what that limit is for.
Bound the limiter's own store call with a timeout — a limiter that hangs turns a slow cache into a slow product regardless of the fail policy.
A read-modify-write split across round trips overshoots by roughly the concurrency, which no sequential test will reveal.
Set the counter's expiry in the same atomic step as its creation, or an interruption between INCR and EXPIRE blocks a caller permanently.
Edge enforcement protects capacity; application enforcement protects fairness and downstream resources. They are layers, not alternatives.

What you will be able to do

Specify all six decisions for a given limit, including an explicitly chosen failure mode
Justify fail-open versus fail-closed for an abuse-prevention limit and a fairness quota
Implement the check as a single atomic operation and explain the overshoot the non-atomic version permits
Place coarse and fine limits at the right layers and say what each layer can and cannot know

The six decisions

Identity, scope, algorithm, storage, coordination — and the failure mode nobody specifies.

Six decisions, and the failure mode nobody specifies

coreintermediate

Designing a rate limiter is six decisions taken in order, and skipping any one of them produces a limiter that works in testing and does something surprising in production. The identity key is what a request is counted against — a user id, an API key, an IP address, a tenant — and it must be something an attacker cannot cheaply change, which is why IP alone is weak and an authenticated identity is strong. The scope is what the limit protects: a whole account, one endpoint, one expensive operation, or the shared resource behind them. The algorithm decides how the count is measured over time. The storage is where counters live, which for anything beyond one process means a shared store. Distributed coordination is how several enforcement points agree, and it always involves a trade between accuracy and the cost of synchronising. And the sixth — the one that is almost never written down — is the failure mode: what the limiter does when its own storage is unavailable. It has exactly two answers, and both are wrong in different situations. Fail open means allow the request, which keeps the product working during a cache outage and removes all protection at the moment the system is least healthy. Fail closed means reject, which preserves protection and turns a limiter-store outage into a full product outage. The right answer differs per limit: a login-attempt limiter protecting against credential stuffing should fail closed, while a generous per-user API quota should fail open. A limiter without a stated failure mode has one anyway — whatever the code does when the store call throws — and it is chosen by accident.

Think of it as

A limiter is a turnstile, and the six decisions are: whose ticket you check, which door it controls, how you count entries over time, where the tally is kept, how several turnstiles keep one tally, and what happens when the tally system goes dark. Every real deployment eventually experiences the last one. Whether the turnstiles then swing free or lock shut is a policy decision that costs money either way, so it is worth making deliberately rather than discovering during the incident.

python

def allow(request):
    key = identity_of(request)          # 1
    limit = limit_for(key, request.route)  # 2, 3
    try:
        return counter.check(key, limit,   # 4, 5
                             timeout=0.02) # bounded
    except StoreUnavailable:
        return limit.fail_open            # 6: stated
                                          # per limit,
                                          # not global

What we're doing: Run the same limiter-store outage against two limits with different failure modes.

limiter-outage.txttext

03:14  the Redis cluster holding all rate-limit
       counters becomes unreachable.

Limit A: login attempts, 5 per account per 15m
  policy: FAIL CLOSED
  effect: every login attempt is rejected with
          429 for 6 minutes.
  cost:   users cannot sign in for 6 minutes.
  benefit: a credential-stuffing run against
          10,000 accounts, in progress at the
          time, is also stopped.

Limit B: API quota, 10,000 requests/hour/key
  policy: FAIL OPEN
  effect: every request is allowed.
  cost:   for 6 minutes, one heavy client could
          exceed its quota. Nothing broke.
  benefit: the API stayed up for everyone.

Same outage. Two policies. Both correct, because
the two limits exist for different reasons.

Now the version with no stated policy:
  the store call raises, the exception propagates,
  and EVERY request guarded by ANY limit returns
  500. The failure mode was "fail closed, loudly,
  with the wrong status code" -- chosen by no one.

6: Failing closed on login is a deliberate six-minute sign-in outage. That is a real cost, accepted because the alternative is removing the only protection against an attack that is more likely during an incident, not less.
14: Failing open here costs at most some over-quota usage by one client for six minutes. The limit exists to keep clients fair to each other, and a fairness rule is not worth an outage.
24: This is what a limiter without a stated failure mode actually does. It is the worst of both policies — protection is lost for anything that would have been allowed, availability is lost for everything else, and the response code tells the caller nothing useful.

Why this works: The failure mode is a property of what each limit is for, not of the limiter implementation, which is why it cannot be set once globally. Writing it down per limit turns an incident's worst moment into a behaviour someone already reasoned about, and gives operators a defensible answer to "why could nobody log in".

Calling the counter store without a timeout

Wrong

python

count = redis.incr(key)      # no timeout
if count > limit:            # a hung Redis makes
    return 429               # every guarded
                             # request hang too

Better

python

try:
    count = redis.incr(key, timeout=0.02)
except (Timeout, StoreUnavailable):
    return limit.fail_open   # decided in advance
if count > limit:
    return 429

What you see: A degraded — not dead — counter store makes every rate-limited endpoint slow rather than failing them fast, so request workers pile up waiting on the limiter and the service exhausts its connection pool. The limiter, added to protect the service, is what takes it down.

Why: A limiter runs on the request path, so its own latency is added to every guarded request. Without a bound, a slow store converts directly into slow requests and then into exhausted capacity — which is why a limiter needs both an explicit timeout and a decision about what to do when that timeout fires.

Six decisions, taken in order

Identity key

hard for an attacker to change

Scope

what is being protected

Algorithm

how the count is measured

Storage

where the counter lives

Coordination

how enforcement points agree

Failure mode

open or closed when the store is down

The six decisions, in the order they are made

The six decisions, in the order they are made
#	Decision	The question it answers
1	Identity key	What is this request counted against, and how hard is that to change?
2	Scope	What is being protected — an account, an endpoint, a downstream resource?
3	Algorithm	How is the count measured over time, and what burst does that permit?
4	Storage	Where does the counter live, and what does that add to the request path?
5	Distributed coordination	How do several enforcement points agree, and how exact must they be?
6	Failure mode	What happens when the counter store is unavailable?

Fail open versus fail closed, per limit

Fail open versus fail closed, per limit
Limit	Failure mode	Why
Login attempts per account	Fail closed	The limiter is the only defence against credential stuffing; losing it during an outage is exactly when it is attacked
Password reset requests	Fail closed	Same reasoning, plus outbound email cost
Per-user API quota (generous)	Fail open	The limit exists for fairness, not safety; rejecting everyone to enforce fairness is a worse outcome
Expensive report generation	Fail closed	The limit protects a scarce downstream resource that an unlimited burst would take down
Public read endpoints behind a CDN	Fail open	The CDN absorbs most traffic; the limiter is a second line, not the only one

Remember: Six decisions in order: identity key (expensive for an attacker to change), scope (what is protected), algorithm, storage, distributed coordination, and failure mode. The sixth is the one that is almost never written down and always exists anyway. Choose it per limit — abuse-prevention limits fail closed, fairness quotas fail open — and always bound the limiter's own store call with a timeout, since a limiter that hangs turns a slow cache into a slow product no matter which policy you picked.

Making it actually limit

Atomic counter operations, the expiry trap, and choosing the layer enforcement runs at.

Atomic counter operations and edge-level enforcement

coreintermediate

Two implementation facts decide whether a limiter actually limits. The first is atomicity. Every limiter is a read-modify-write — read the count, compare it to the limit, write the new count — and if those are three separate round trips to a shared store, concurrent requests interleave between them and the limit is exceeded by roughly the number of requests in flight. The fix is to make the whole check one operation the store executes indivisibly: an atomic increment that returns the new value, or a small server-side script that reads, decides and writes without anything else running in between. A token bucket, which needs to compute how many tokens have refilled since the last call, cannot be expressed as a single increment and is the standard reason to use a script. The second fact is where enforcement happens. A limit enforced inside your application has already cost you a TLS handshake, a load-balancer hop, a connection from the pool and a request worker before it rejects anything — which means an attacker can still saturate your capacity while being perfectly rate-limited. Enforcing at the edge, in a CDN or gateway, rejects the request before it reaches your infrastructure at all. The two are layered rather than alternatives: coarse, cheap limits at the edge protect capacity, and fine-grained, identity-aware limits in the application enforce the rules that need to know who the caller is and what they are asking for.

Think of it as

Two failures of a bouncer. The first is a bouncer who counts by looking at a clipboard, writing a new number and putting the clipboard down — with three other bouncers doing the same thing at other doors. The count is wrong in proportion to how busy the night is, and the fix is one shared tally that only one hand can touch at a time. The second is putting the bouncer at the bar rather than at the street door: everyone gets inside, fills the room and is then turned away at the counter. The room is full either way. Rate limiting has both failures, and they need different fixes.

text

-- one atomic operation: read, decide, write
-- (Redis executes a script without interleaving)
local current = redis.call('INCR', KEYS[1])
if current == 1 then
  redis.call('EXPIRE', KEYS[1], ARGV[2])  -- same
end                                        -- step
if current > tonumber(ARGV[1]) then
  return 0        -- reject
end
return 1          -- allow

What we're doing: Measure the overshoot a non-atomic limiter permits, then remove it.

overshoot.txttext

Limit: 100 requests per minute per API key.
Load:  40 concurrent requests arrive at 3 app
       instances at the same instant, when the
       counter reads 98.

NON-ATOMIC  (GET, compare, SET)
  all 40 requests GET   -> 98
  all 40 compare 98 < 100 -> allow
  all 40 SET 99
  Result: 138 requests allowed against a limit
  of 100, and the counter now reads 99 rather
  than 138 -- so the next minute starts wrong
  as well.

ATOMIC INCREMENT
  the store serialises 40 increments -> the
  values 99..138 are handed out, one per caller
  requests receiving 99, 100        -> allowed
  requests receiving 101 and above  -> rejected
  Result: exactly 100 allowed.

The overshoot is not proportional to traffic
volume -- it is proportional to CONCURRENCY,
which is why it is invisible in testing and
appears at exactly the load the limit exists
to handle.

8: Every one of the forty requests reads the same value, because none of them has written yet. The comparison is correct for each request individually and wrong for all of them together, which is why the bug survives code review.
13: The counter being left at 99 rather than 138 is the second, quieter half of the bug: the lost updates mean the following window also starts from a wrong value, so the limiter under-counts continuously under sustained concurrency.
24: This is the reason to be suspicious of a limiter that has never been load-tested concurrently. A sequential test at any volume passes, because sequential requests never interleave.

Why this works: The gap between the read and the write is the entire bug, and it cannot be closed with more careful application code — only by moving the decision into an operation the store performs indivisibly. That is also why a token bucket, whose decision needs refill arithmetic rather than a bare increment, is implemented as a server-side script rather than as several commands.

INCR and EXPIRE as two separate commands

Wrong

text

INCR   ratelimit:key   -> 1
# a crash, a network drop, or a failover here
EXPIRE ratelimit:key 60
# the key now lives forever; this caller is
# rate-limited permanently, with no way for the
# counter to reset

Better

text

-- both in one script, so either both run or
-- neither does
local n = redis.call('INCR', KEYS[1])
if n == 1 then redis.call('EXPIRE', KEYS[1], 60) end
return n

What you see: A small number of users are permanently rate-limited, and clearing the key fixes it until it happens again. It correlates with store failovers or deploys rather than with the affected users doing anything unusual.

Why: A counter with no expiry never resets, so the window never rolls over and the caller stays blocked indefinitely. Two commands can be interrupted between them; one script cannot, which turns "the key always has a TTL" from a hope into a property of the operation.

Enforcement layers, outermost first

CDN / edge

coarse limits by IP and path — rejects before any of your capacity is consumed

API gateway

per-key and per-route limits — protects application capacity and pools

Application

per-user, per-plan, per-operation limits — knows everything, costs the most to reach

Downstream resource

the scarce thing all of the above exist to protect

Three ways to implement the check, and what each permits

Three ways to implement the check, and what each permits
Implementation	Round trips	Worst-case overshoot
GET, compare in app, SET	2 (plus app logic between)	Roughly the number of concurrent requests
Atomic increment returning the new value	1	None for fixed-window counting
Server-side script (read, decide, write)	1	None; also supports token-bucket refill arithmetic

Where to enforce, and what each layer can and cannot know

Where to enforce, and what each layer can and cannot know
Layer	Knows	Protects	Cannot do
CDN / edge	IP, path, coarse headers	Your entire infrastructure, before any cost is paid	Per-user or per-plan limits requiring authentication
API gateway	API key, route, tenant	Application capacity, connection pools	Limits that depend on request body or business state
Application	Everything — user, plan, resource, cost of the operation	Downstream resources and fairness between callers	Anything, without first paying the full request cost

Remember: Make the check one atomic operation — an increment that returns the new value, or a server-side script for anything needing refill arithmetic — because a read-modify-write split across round trips overshoots by roughly the concurrency, which is invisible in sequential tests. Set the expiry in the same atomic step, or a crash between INCR and EXPIRE blocks a caller forever. And layer enforcement: coarse limits at the edge protect capacity before any cost is paid, fine identity-aware limits in the application enforce product rules.