beginnerintermediateadvanced

System Design · Section 12

Load Balancing

Level: intermediate
Read: 20 min
Concepts: 4

A load balancer decides which backend instance handles each request — at the transport layer (L4, IP/port only) or the application layer (L7, the full HTTP request), using an algorithm from plain round robin to consistent hashing, backed by health checks and connection draining to keep the pool honest. All of it works best, and horizontal scaling becomes simple, when the instances being balanced across are stateless.

This section

Load Balancing and Traffic Routing Strategies Simplified | System Design Fundamentals 2026SCALER

What is true here

L4 routes by IP/port only and is fast; L7 reads the HTTP request and can route by path, header or cookie.
Round robin, weighted round robin, least connections, hashing and consistent hashing are five distinct ways to pick an instance.
Health checks decide pool membership; connection draining retires an instance without dropping in-flight requests.
Sticky sessions pin a client to an instance for cache locality — never as the only place session state lives.
Stateless instances are what let a load balancer route any request to any instance, which is what makes horizontal scaling simple.

What you will be able to do

Choose L4 or L7 for a given routing requirement
Pick an appropriate load-balancing algorithm for a given traffic shape
Explain how health checks, draining and failover keep a pool of instances honest
Explain why statelessness is the precondition for easy horizontal scaling

Routing decisions

Which layer a balancer operates at, and the algorithms it uses to pick an instance.

L4 vs L7 load balancing

coreintermediate

An L4 (transport-layer) load balancer routes by IP and port alone, without looking inside the request — fast, but blind to content. An L7 (application-layer) load balancer reads the actual HTTP request — path, headers, cookies — and can route on that, at the cost of more work per request.

Think of it as

An L4 balancer is like a mail sorting machine that only reads the zip code on the envelope — fast, but it has no idea what is inside. An L7 balancer is like a person who opens the envelope, reads the letter, and routes it based on what it actually says — slower per item, but able to make far more precise decisions.

text

L4: route by (source IP, dest IP, port) — no payload inspection
L7: route by (HTTP method, path, host, header, cookie)

What we're doing: Show the same traffic routed differently by an L4 balancer versus an L7 balancer.

l4-vs-l7-routing.txttext

Two requests arrive on port 443:
  GET /api/orders/42
  GET /static/app.js

L4 load balancer (TCP-level):
  Both connections look identical — same port, same
  protocol. Routed to whichever backend is next in
  the pool, with no distinction between the two paths.

L7 load balancer (HTTP-level):
  Reads the path after establishing the connection:
    /api/*    → orders-service pool (5 instances)
    /static/* → static-asset pool (2 instances)
  Two different requests, two different destinations.

5: L4 cannot distinguish the two requests — it never looks past the IP/port.
10: L7 parses the actual HTTP request to make a routing decision the L4 balancer structurally cannot make.

Why this works: Content-based routing (API traffic to one pool, static assets to another, or A/B testing by cookie) requires an L7 balancer — an L4 balancer physically does not have access to that information.

Expecting an L4 load balancer to route by URL path

Wrong

text

// L4 balancer config, expecting path-based routing
route: { path: '/api/*', pool: 'api-pool' }
// L4 has no concept of "path" — this silently
// does nothing

Better

text

// use an L7 balancer/reverse proxy for path routing
// L4 balancer only sees IP:port — route by port
// or use a separate L7 layer in front of it

What you see: Path-based routing rules configured on an L4 balancer are silently ignored or rejected at config time, because L4 never parses the HTTP request to find a path in the first place.

Why: The two layers see fundamentally different data. Choosing the right layer for the job is a prerequisite, not an implementation detail.

Same traffic, two routing decisions

L4 (transport layer)

+Sees only IP + port
+Cannot tell /api/orders from /static/logo.png
+Forwards any packet on port 443 to any backend IP
+Very low cost per request

L7 (application layer)

−Reads the full HTTP request
−Routes /api/* to the orders-service pool
−Routes /static/* to the CDN origin pool
−Higher cost — parses the request

What each layer can see and decide on

What each layer can see and decide on
Layer	Sees	Can route on	Cost per request
L4 (transport)	IP + port	Source/destination IP, port	Very low
L7 (application)	Full HTTP request	Path, host header, cookie, body	Higher (parses the request)

Together

text

L4 load balancer: forwards any packet on port 443
  to one of the backend IPs — same decision for
  /api/orders and /static/logo.png.

L7 load balancer: reads the HTTP request and routes
  /api/*    → orders-service pool
  /static/* → CDN origin pool
  by path, something an L4 balancer cannot do.

Remember: L4 routes by IP/port only, fast and content-blind; L7 reads the actual HTTP request and can route by path, header or cookie, at higher cost per request.

Load balancing algorithms

coreintermediate

A load balancer needs a rule for picking which backend instance handles the next request. Round robin cycles through instances evenly; weighted round robin favors bigger instances; least connections sends traffic to whichever instance is least busy right now; hashing routes the same client (or key) to the same instance consistently.

Think of it as

Picture checkout lines at a store. Round robin sends the next customer to the next register in turn, regardless of how full each line already is. Least connections sends them to whichever line is shortest right now. Weighted round robin gives a faster cashier more customers than a slower one. Hashing always sends a specific loyalty-card number to the same cashier, so that cashier remembers their preferences.

text

round-robin:        instance[i % N]
weighted round-robin: instance chosen proportional to weight
least-connections:   instance with min(open_connections)
hashing:             instance[hash(key) % N]
consistent hashing:  instance nearest to hash(key) on a ring

What we're doing: Show why plain hashing reshuffles almost every client on scale-up, and how consistent hashing avoids it.

consistent-hashing-scaleup.txttext

Plain hash, 3 instances: hash(client_id) % 3

  client A → hash=17 → 17 % 3 = 2 → instance 2
  client B → hash=22 → 22 % 3 = 1 → instance 1
  client C → hash=9  → 9  % 3 = 0 → instance 0

Scale to 4 instances: hash(client_id) % 4

  client A → 17 % 4 = 1 → instance 1  (moved!)
  client B → 22 % 4 = 2 → instance 2  (moved!)
  client C → 9  % 4 = 1 → instance 1  (moved!)

Nearly every client's cache/session on the old
instance is now cold on a different instance.
Consistent hashing places instances on a ring so
only the clients between the new instance and its
neighbor move — everyone else stays put.

9: All three example clients moved to a different instance after adding just one more — a full reshuffle.
15: This is exactly the problem consistent hashing is built to avoid — see the mistake below.

Why this works: A cache or sticky session tied to "whichever instance a client hashes to" becomes worthless across a scaling event if the hashing scheme reshuffles nearly everyone — consistent hashing keeps most assignments stable.

Using plain modulo hashing for a system that scales up and down often

Wrong

text

instance = pool[hash(key) % pool.length]

Better

text

// consistent hashing: place instances and keys
// on a ring, route to the nearest instance
// clockwise — adding/removing one instance only
// reassigns its immediate neighbors' keys

What you see: Every scale-up or scale-down event invalidates almost all per-instance caches or sticky sessions at once, because `% pool.length` changes nearly every key's target instance whenever the pool size changes.

Why: Modulo hashing ties every key's assignment to the total instance count. Consistent hashing decouples most assignments from the total count, so scaling events only disturb a small fraction of keys.

Adding a 4th instance: plain hashing vs consistent hashing

Plain hashing (% N)

+hash(key) % 3 -> hash(key) % 4
+Almost every client's assignment changes
+Caches and sticky sessions go cold at once

Consistent hashing

−Instances placed on a ring
−Only keys between old and new neighbor move
−Most assignments stay untouched

Choosing an algorithm

Choosing an algorithm
Algorithm	Decision basis	Good fit when
Round robin	Fixed rotation order	Instances are equal-sized, requests are similar cost
Weighted round robin	Rotation + a weight per instance	Instances have different capacity
Least connections	Current open-connection count	Requests vary a lot in how long they take
Hashing	Hash of a key (IP, session)	The same client should keep hitting the same instance
Consistent hashing	Hash placed on a ring of instances	Instances scale up/down often, minimize reshuffling

Together

text

3 instances, plain hashing on client IP:
  hash(ip) % 3  → picks instance 0, 1, or 2

Add a 4th instance:
  hash(ip) % 4  → almost every client's assignment
  changes, even though only one instance was added.

Consistent hashing places instances on a ring instead:
  adding instance 4 only reassigns the clients that
  land between instance 3 and instance 4 on the ring —
  everyone else's assignment is untouched.

Remember: Round robin/weighted round robin ignore current load; least connections adapts to it; hashing pins a key to an instance; consistent hashing does that without reshuffling everyone on scale events.

Keeping the pool healthy

The operational mechanics beyond an algorithm, and why statelessness makes all of it simpler.

Health checks, connection draining, sticky sessions and failover

standardintermediate

A load balancer needs more than an algorithm — it needs to know which instances are actually healthy (health checks), how to retire one without dropping in-flight requests (connection draining), how to keep a client on the same instance when needed (sticky sessions), and what to do when an instance dies mid-request (failover).

Think of it as

A load balancer is like a dispatcher for a fleet of delivery drivers. It periodically calls each driver to check they are still working (health check), tells a driver going off shift to finish their current delivery before stopping (connection draining), sometimes assigns the same customer to the same driver every time for familiarity (sticky sessions), and immediately reroutes a delivery if a driver's van breaks down mid-route (failover).

text

health check:       GET /healthz every N seconds
                     fail K times → remove from pool
connection draining: stop new traffic, wait up to T
                     seconds for in-flight requests
sticky session:      cookie or IP hash → same instance
failover:            instance dies → route future
                     requests to remaining healthy ones

What we're doing: Show a rolling deployment relying on health checks and connection draining to avoid dropping requests.

rolling-deploy-drain.txttext

Deploying a new version across 6 instances, one at a time:

1. Load balancer marks instance 1 for draining:
   - stops sending it new requests
   - lets its current in-flight requests finish
   - waits up to 30s, then the deploy tool restarts it

2. Once restarted, instance 1 must pass 2 consecutive
   health checks (GET /healthz → 200 OK) before the
   load balancer adds it back to the pool.

3. Repeat for instances 2 through 6, one at a time —
   at every moment, at least 5 of 6 instances are
   serving live traffic.

4: Draining, not killing — a request already in flight is allowed to finish instead of being dropped.
9: Requiring 2 consecutive passing checks (not 1) avoids flapping an instance back in right as it restarts and briefly looks healthy.
12: One instance at a time is what keeps the deploy from ever taking capacity below 5/6.

Why this works: Without draining, a deploy or scale-down kills in-flight requests mid-response; without health checks, a broken instance keeps receiving traffic it cannot serve.

Killing an instance the moment a deploy starts, with no draining

Wrong

text

// deploy script
stopInstance(instance1);  // in-flight requests
                          // get connection-reset errors
startNewVersion(instance1);

Better

text

// deploy script
loadBalancer.drain(instance1, { timeout: 30 });
// wait for in-flight requests to finish or timeout
stopInstance(instance1);
startNewVersion(instance1);

What you see: A subset of users see connection-reset or 502 errors during every deploy, exactly correlated with the deploy timing — because in-flight requests were killed rather than allowed to finish.

Why: Draining is what separates "stop sending new work" from "kill everything running right now." Skipping it turns every routine deploy into a small, avoidable outage for whoever had a request in flight.

One backend instance's life in the pool

In rotation (healthy)

start

Draining (retiring)

Removed from pool

Failed health check

Remember: Health checks decide who is in the pool; draining retires an instance without dropping in-flight work; sticky sessions pin a client to an instance for locality, not as the source of truth; failover reroutes when one dies.

Why stateless servers make horizontal scaling easier

standardintermediate

If an application instance holds no local state (no data that only it has), a load balancer can route any request to any instance interchangeably. That is what makes horizontal scaling — adding more instances — simple: a new instance is immediately as capable as any existing one.

Think of it as

Think of interchangeable cashiers versus cashiers who each keep their own private notebook of a customer's order history. Interchangeable cashiers can be added or removed freely — any of them can serve any customer. A cashier with a private notebook cannot be swapped out without losing information, so the store is stuck routing that customer back to the same person.

text

stateless: any_instance.handle(request) — result is
           the same regardless of which instance runs it
stateful:  request must reach the specific instance
           holding the relevant local state

What we're doing: Contrast scaling a stateless instance pool against a pool where each instance holds local session state.

stateless-scaling.txttext

Stateless pool (session data in shared Redis):
  Traffic doubles → add 3 more instances.
  Load balancer immediately routes to all 9 instances.
  Every instance reads/writes the same shared Redis —
  no instance is "special."

Stateful pool (session data in each instance's memory):
  Traffic doubles → add 3 more instances.
  The 3 new instances have never seen any existing
  session — a request for an existing session MUST
  reach one of the original 6, or the session is gone.
  Scaling requires sticky routing to the ORIGINAL
  instance, and the new instances can only take
  brand-new sessions.

5: Any of the 9 instances can serve any request — this is the entire benefit.
12: The new instances are only useful for new sessions — half the benefit of scaling up is lost.

Why this works: Horizontal scaling assumes a new instance is immediately as useful as an existing one. Local state breaks that assumption, because the new instance did not inherit any of it.

Storing session or upload state in an instance's local memory or disk

Wrong

text

// in-process map, only this instance's memory
const sessions = new Map();
sessions.set(sessionId, userData);

Better

text

// shared store any instance can read
await redis.set(`session:${sessionId}`, userData);

What you see: A user gets logged out or loses their in-progress upload whenever the load balancer happens to route their next request to a different instance than the one that handled their last one.

Why: Local, in-memory state ties a client to one specific instance. Moving it to a shared store (this section's Stateless Architecture principle) is what lets any instance serve any request, which is the precondition for scaling horizontally without special-casing routing.

Adding capacity: stateless vs stateful instances

Stateless pool

+Session data lives in shared Redis, not on any one instance
+A new instance is immediately as capable as any existing one
+Load balancer routes to any healthy instance, no exceptions
+Traffic doubles → add instances → done

Stateful pool

−Each instance holds its own local session data
−A request must reach the specific instance holding it
−A new instance starts with no data — cannot serve existing sessions
−Scaling requires sticky routing or a state-migration plan

Remember: A stateless instance can serve any request, which is what lets horizontal scaling add capacity instantly instead of needing to replicate state onto every new instance first.