beginnerintermediateadvanced

System Design · Section 6

Latency and Throughput

Level: intermediate
Read: 22 min
Concepts: 5

The two axes every performance conversation needs to separate — latency, the time for one operation, and throughput, work completed per unit time — measured precisely with percentiles (p50 through p99.9) rather than an average that dilutes away the slow outliers users actually feel, connected by Little's Law (concurrency ≈ throughput × latency), and shaped in practice by which portions of a request path are genuinely serial versus safe to run in parallel.

This section

Latency vs Throughput | System Design EssentialsBe A Better Dev

What is true here

Latency = time for one operation; throughput = operations completed per unit time.
p50 is the median; p90/p95/p99/p99.9 progressively expose the tail.
An average dilutes a few very slow requests across many fast ones — it can look healthy while a real tail exists.
Little's Law: concurrency ≈ throughput × latency — sizes worker/connection pools and explains why latency regressions cap throughput.
Serial steps sum their latencies; independent steps run in parallel and cost only the slowest one.

What you will be able to do

Distinguish latency from throughput and know what raises each independently
Read and choose an appropriate latency percentile (p50 through p99.9) for a given use
Explain why an average latency figure can hide a real tail-latency problem
Apply Little's Law to size a worker or connection pool from a target throughput and measured latency
Identify which calls in a request path are genuinely serial and which are safe to parallelize

The two quantities

What latency and throughput each measure, and the percentiles used to report latency precisely.

Latency vs throughput

corebeginner

Latency is how long one request takes, end to end. Throughput is how many requests the system completes per unit of time. They are different axes — a system can raise one while lowering the other.

Think of it as

Think of a highway. Latency is how long one car takes to drive it end to end. Throughput is how many cars pass a checkpoint per hour. Adding lanes raises throughput without changing any single car's travel time; a faster speed limit lowers latency without necessarily changing how many cars fit through. The two are related but not the same lever.

text

latency    = time for ONE operation (ms, s)
throughput = operations completed / unit time (req/s)

raising concurrency raises throughput without
necessarily changing latency

What we're doing: Show latency and throughput moving independently as concurrency changes.

latency-vs-throughput.txttext

A single-threaded server: 50ms per request.

  1 request at a time:
    latency = 50ms, throughput = 20 req/s

  10 requests handled concurrently (10 workers):
    latency per request ≈ 50ms (unchanged)
    throughput = 10 workers × 20 req/s = 200 req/s

  Same 10 workers, but a slow downstream call
  raises latency to 200ms per request:
    throughput = 10 workers / 0.200s = 50 req/s

3: One worker, sequential: throughput is exactly the inverse of latency.
6: Ten workers in parallel multiply throughput by ten while each request still takes the same 50ms.
10: When per-request latency rises, throughput falls even with the same worker count — the two move together only when concurrency is fixed.

Why this works: Confusing the two leads to the wrong fix: adding servers does not help a single slow request, and speeding up one code path does not raise the ceiling on total requests per second if concurrency is capped.

Reporting throughput as if it describes user experience

Wrong

text

"The system handles 200 req/s, so it's fast."

Better

text

"The system handles 200 req/s, and p95 latency
per request is 180ms."

What you see: A dashboard shows healthy throughput while users report the app feels slow — because throughput says nothing about how long any individual user waited.

Why: Throughput is a system-wide aggregate; latency is what any single user actually experiences. A system can hit a high throughput number while every request is unpleasantly slow, if enough of them run in parallel.

One highway, two different measurements

Latency

+Time for ONE car to drive the highway end to end
+Measured in ms or s
+Lowered by a faster route, fewer hops

Throughput

−Cars passing a checkpoint per hour
−Measured in req/s, ops/s
−Raised by adding lanes (parallel capacity)

Latency vs throughput

Latency vs throughput
Quantity	Answers	Typical unit	Raised by
Latency	"How long did my request take?"	ms, s	faster code path, fewer hops, less queuing
Throughput	"How much work got done?"	req/s, ops/s	more parallel capacity, batching, more workers

Together

text

One API server, one request at a time:
  latency = 50ms per request
  throughput = 1 / 0.050s = 20 req/s

Same server, 10 requests handled concurrently:
  latency per request ≈ still ~50ms
  throughput ≈ 10 / 0.050s = 200 req/s

Remember: Latency = time for one operation; throughput = operations per unit time. More parallel capacity raises throughput, not latency.

See also: percentile latency · littles law

Percentile latency: p50, p90, p95, p99, p99.9

corebeginner

A percentile latency says what fraction of requests were at or below a given time. p50 is the median; p99 is the time the slowest 1% of requests exceed. Higher percentiles reveal the worst experiences a fast-looking median can hide.

Think of it as

Sort every request's latency from fastest to slowest and line them up. p50 is the value halfway along that line. p99 is the value 99% of the way along — only the slowest 1% of requests are worse than it. A service can have a great p50 and a terrible p99 at the same time, because the median only describes the middle of the line, not its tail.

text

pN = the latency value at or below which N% of
     requests fall, after sorting all latencies
     fastest to slowest

p50 = median · p99 = slowest 1% starts here

What we're doing: Show how the same request stream produces a fast p50 and a slow p99, and why that gap matters.

percentile-latency.txttext

An API endpoint, 10,000 requests measured:

  p50:   35ms   (typical request, cache hit)
  p90:   80ms   (typical request, cache miss)
  p95:  150ms   (cache miss + light contention)
  p99:  900ms   (cache miss + DB connection wait)
  p99.9: 4,000ms (cache miss + DB wait + GC pause)

At 1,000,000 requests/day, p99.9 alone means
~1,000 requests a day take 4+ seconds.

3: p50 looks great — most users never notice a delay.
7: p99 is 25x the median — the slowest 1% wait almost a full second, from causes the median never sees.
10: At real scale, even a 0.1% tail is thousands of unhappy users a day — the percentile that looked like a rounding error is an incident.

Why this works: A single average or median hides exactly the requests most likely to cause complaints, timeouts, or retries — the tail. Percentiles make the shape of that tail visible instead of averaging it away.

Reporting only the average or median latency

Wrong

text

"Average latency is 45ms — the API is fast."

Better

text

"p50 is 35ms, but p99 is 900ms — 1% of
requests are 25x slower than typical."

What you see: The dashboard says the API is fast, but support tickets keep arriving about timeouts — because the 1% tail causing them never shows up in an average.

Why: An average is dominated by the common case and can look fine even when a meaningful fraction of requests are badly slow. Percentiles, especially p99 and above, are what actually describe the worst experiences users have.

10,000 requests, sorted fastest to slowest

SlowFast

Median caseDeep tail

p50: 35ms — cache hit
p90: 80ms — cache miss
p95: 150ms — miss + contention
p99: 900ms — miss + DB wait
p99.9: 4,000ms — + GC pause

Percentile latency

Percentile latency
Percentile	Meaning	Typical use
p50 (median)	half of requests are at or below this value	general "typical" latency
p90	90% of requests at or below this value	everyday performance target
p95	95% of requests at or below this value	common SLO target
p99	99% of requests at or below this value	tail-latency SLO, worst-case-ish UX
p99.9	99.9% of requests at or below this value	high-scale systems where 0.1% is still thousands of users

Together

text

1,000 sorted request latencies (ms), fastest to slowest:

  p50  -> the 500th value   -> 40ms
  p90  -> the 900th value   -> 120ms
  p95  -> the 950th value   -> 180ms
  p99  -> the 990th value   -> 900ms
  p99.9 -> the 999th value  -> 4,200ms

Remember: pN = the latency N% of requests fall at or below. p50 = median; p99/p99.9 expose the tail an average hides.

Why averages hide tail latency problems

standardbeginner

An average is easily dominated by the common, fast requests — a small number of very slow outliers barely move it, even though every one of those outliers is a real user having a bad experience.

Think of it as

Average 1,000 latencies where 990 are 20ms and 10 are 5,000ms: the mean lands around 70ms — looking almost fine — while ten real users waited five full seconds. The average is a single number computed across every value, so a few extreme values get diluted by the much larger group of ordinary ones. Percentiles do not have this problem because they report an actual value at a position in the sorted list, not a blend.

text

mean = sum(all latencies) / count

a few very slow requests barely move the mean,
because they're divided across a much larger count
of fast ones — the mean dilutes outliers

What we're doing: Compute a mean and a p99 from the same latency sample and show how far apart they land.

averages-hide-tails.txttext

1,000 requests: 990 at 20ms, 10 at 5,000ms.

  mean = (990 × 20 + 10 × 5,000) / 1,000
       = (19,800 + 50,000) / 1,000
       = 69.8ms

  p99  = the 990th value in sorted order = 20ms
  p99.9 = the 999th value = 5,000ms

The mean (69.8ms) looks mildly slow. p99.9 shows
the real story: the slowest 0.1-1% wait 250x longer
than everyone else.

3: The mean blends 990 fast values with 10 very slow ones into one number.
8: p99 in this sample is still 20ms — the slow 1% sits right at the percentile boundary and barely nudges it.
9: p99.9 is where the outliers actually surface: a value 250x the fast baseline.

Why this works: A mean is a single number that any distribution shape can produce — a system with occasional severe slowdowns and a system with uniformly mediocre latency can report the same mean while feeling completely different to users. Percentiles distinguish the two.

One sample, five statistics — 990 requests at 20ms, 10 at 5,000ms

Computed from the stated sample, not measured. mean = (990 x 20 + 10 x 5,000) / 1,000 = 69.8ms. p99 is the 990th value in sorted order, still 20ms; p99.9 is the 999th, 5,000ms. Four bars are almost invisible beside the fifth, and that is the point — every statistic up to p99 reports a healthy service while ten users wait five seconds.

Remember: A mean dilutes a few very slow requests across many fast ones — it can look healthy while a real tail of bad experiences exists underneath it.

Reasoning about a request path

The law connecting concurrency to throughput and latency, and where a request path's total latency actually comes from.

Little's Law: concurrency ≈ throughput × latency

coreintermediate

Little's Law connects the two quantities this section opened with: the number of requests being handled at once (concurrency) equals throughput multiplied by latency. Raise either throughput or latency and required concurrency rises with it.

Think of it as

Picture a checkout line: throughput is how many customers arrive per minute, latency is how long each one takes to check out, and concurrency is how many customers are in line (including being served) at any moment. If customers arrive faster, or each one takes longer, more customers end up in the line simultaneously — that is Little's Law. It tells you how many concurrent workers, connections, or threads a system needs to sustain a given throughput at a given latency.

text

concurrency = throughput (req/s) × latency (s)

rearranged:
throughput = concurrency / latency
latency    = concurrency / throughput

What we're doing: Use Little's Law to size a worker pool for a target throughput, then show what happens if latency rises unexpectedly.

littles-law.txttext

Target: sustain 1,000 req/s.
Average request latency: 150ms (0.15s).

  concurrency needed = 1,000 × 0.15 = 150

Provision a worker pool of ~150 (plus headroom).

Now a downstream dependency slows to 600ms (0.6s):

  concurrency needed = 1,000 × 0.6 = 600

The same 150-worker pool can now sustain only:
  throughput = 150 / 0.6 = 250 req/s

Requests queue, and effective throughput drops
to a quarter of the target.

4: The pool is sized directly from the target throughput and the measured latency — not guessed.
9: A slower downstream call raises latency 4x, which raises required concurrency 4x for the same throughput.
12: With concurrency capped at the original pool size, the system can only sustain a quarter of the target throughput — the rest queues or times out.

Why this works: Little's Law is what turns "we need to handle 1,000 req/s" into an actual worker or connection pool size — and explains why a slow downstream dependency degrades throughput even when nothing about request volume changed.

Sizing a connection pool from throughput alone, ignoring latency

Wrong

text

"We need 1,000 req/s, so provision 1,000
connections."

Better

text

"We need 1,000 req/s at 150ms average latency,
so concurrency = 1,000 × 0.15 = 150 connections."

What you see: A pool sized to match throughput 1:1 is wildly oversized when latency is low, and still runs out under load if latency spikes — because the actual required size depends on both numbers together, not throughput alone.

Why: Concurrency, not raw throughput, is what a pool or worker count actually needs to match. Little's Law is the formula connecting the two, and skipping it means guessing a pool size instead of computing it.

A downstream slowdown caps throughput, at a fixed pool size

Target throughput

1,000 req/s

Latency rises

150ms -> 600ms

Concurrency needed

1,000 × 0.6 = 600

Fixed pool (150)

caps actual throughput at 250 req/s

Little's Law in practice

Little's Law in practice
Given	Formula	Worked example
Throughput + latency	concurrency = throughput × latency	500 req/s × 0.2s = 100 concurrent requests
Concurrency + latency	throughput = concurrency ÷ latency	100 workers ÷ 0.2s = 500 req/s max
Concurrency + throughput	latency = concurrency ÷ throughput	100 ÷ 500 req/s = 0.2s average latency

Together

text

A service handles 500 req/s at 200ms average latency:

  concurrency = 500 req/s × 0.2s = 100

The connection pool / worker pool needs at least
~100 slots to sustain that throughput at that latency
without requests queuing.

Remember: concurrency ≈ throughput × latency — raising either raises how many requests must be in flight at once.

Serial vs parallel portions of a request path

standardintermediate

A request often calls several downstream services. Steps that must happen one after another (serial) add their latencies together; steps that can run at the same time (parallel) only cost the slowest one. Total latency depends on which shape the path actually has.

Think of it as

If step B needs the result of step A, they are serial and their latencies sum. If two steps are independent of each other — fetching a user profile and fetching their recent orders, say — they can run in parallel, and the pair only costs as much as the slower of the two. Most real request paths are a mix: some serial dependencies, some independent calls that could run in parallel but a naive implementation runs serially anyway.

text

serial:   total = sum(step latencies)
parallel: total = max(step latencies)

a request path is usually a mix of both

What we're doing: Reshape one request path from fully serial to serial-plus-parallel and show the latency drop.

serial-vs-parallel.txttext

A product page loads four things:
  auth check:      20ms  (must happen first)
  product details:  50ms  (needs auth)
  reviews:          80ms  (needs auth, not product)
  recommendations: 100ms  (needs auth, not product/reviews)

All four serial:
  20 + 50 + 80 + 100 = 250ms

Auth first, then details/reviews/recommendations
in parallel (none of the three depend on each other):
  20 + max(50, 80, 100) = 120ms

8: Running every step one after another adds all four latencies — 250ms, even though three of the four have no dependency on each other.
12: Auth genuinely has to happen first, but the other three are independent — running them in parallel drops total latency to the slowest single call.

Why this works: A request path's total latency depends entirely on which calls actually depend on each other. Serializing independent calls is one of the most common, and most fixable, sources of unnecessary latency in a design.

Serializing calls that have no dependency on each other

Wrong

text

await fetchProductDetails()
await fetchReviews()
await fetchRecommendations()
// 50 + 80 + 100 = 230ms, run one after another

Better

text

await Promise.all([
  fetchProductDetails(),
  fetchReviews(),
  fetchRecommendations(),
])
// max(50, 80, 100) = 100ms, run concurrently

What you see: A page takes noticeably longer to load than any individual data source would suggest, because independent calls are written sequentially in the code even though nothing requires that order.

Why: Code is often written serially by default, regardless of whether the underlying calls actually depend on each other. Identifying the real dependency graph — not the order the code happens to be written in — is what reveals where parallelism is available.

Same four calls, two shapes

Bars are drawn to scale in milliseconds. Only the auth call has a real dependency; the code order created the other three.

Serial vs parallel cost

Serial vs parallel cost
Shape	Total latency	Example
Serial (A then B)	latency(A) + latency(B)	authenticate, then fetch the authorized resource
Parallel (A and B)	max(latency(A), latency(B))	fetch profile and fetch orders at the same time

Together

text

Auth check: 20ms. Profile fetch: 40ms. Orders fetch: 60ms.

Serial (auth -> profile -> orders):
  20 + 40 + 60 = 120ms

Auth must come first, but profile and orders
are independent once authenticated:
  20 + max(40, 60) = 80ms

Remember: Serial steps sum their latencies; independent steps run in parallel and only cost the slowest one. Identify real dependencies, not code order.

See also: littles law · averages hide tails