beginnerintermediateadvanced

System Design · Latency and Throughput

Latency and Throughput

Two axes, five percentiles, one law connecting them, and where a request path actually spends its time

Latency vs throughputtime per op vs ops per second
p50 to p99.9the median vs the tail
Little's Lawconcurrency ≈ throughput × latency
Serial vs parallelindependent calls only cost the slowest one

1.Latency vs throughput

Latency is how long one request takes, end to end. Throughput is how many requests the system completes per unit of time. They are different axes — a system can raise one while lowering the other.

Key features

Latency: time for one operation, start to finish — measured in ms or s
Throughput: completed operations per unit time — measured in req/s, ops/s
Adding parallel capacity (more servers, more lanes) raises throughput, not latency
A faster single operation (a quicker query, less network hop) lowers latency, not necessarily throughput
A design can optimize for one at the expense of the other — batching raises throughput but adds latency per item

Latency vs throughput

Latency vs throughput
Quantity	Answers	Typical unit	Raised by
Latency	"How long did my request take?"	ms, s	faster code path, fewer hops, less queuing
Throughput	"How much work got done?"	req/s, ops/s	more parallel capacity, batching, more workers

One highway, two different measurements

Latency

Time for ONE car to drive the highway end to end
Measured in ms or s
Lowered by a faster route, fewer hops

Throughput

Cars passing a checkpoint per hour
Measured in req/s, ops/s
Raised by adding lanes (parallel capacity)

Command

text

latency    = time for ONE operation (ms, s)
throughput = operations completed / unit time (req/s)

raising concurrency raises throughput without
necessarily changing latency

Latency = time for one operation; throughput = operations per unit time. More parallel capacity raises throughput, not latency.

2.Percentile latency: p50, p90, p95, p99, p99.9

A percentile latency says what fraction of requests were at or below a given time. p50 is the median; p99 is the time the slowest 1% of requests exceed. Higher percentiles reveal the worst experiences a fast-looking median can hide.

Key features

p50 (median): half of requests are faster, half slower
p90: 90% of requests are at or below this latency
p95: 95% of requests are at or below this latency
p99: only the slowest 1% of requests exceed this latency
p99.9: only the slowest 0.1% of requests exceed this latency — the deep tail

Percentile latency

Percentile latency
Percentile	Meaning	Typical use
p50 (median)	half of requests are at or below this value	general "typical" latency
p90	90% of requests at or below this value	everyday performance target
p95	95% of requests at or below this value	common SLO target
p99	99% of requests at or below this value	tail-latency SLO, worst-case-ish UX
p99.9	99.9% of requests at or below this value	high-scale systems where 0.1% is still thousands of users

10,000 requests, sorted fastest to slowest

SlowFast

Median caseDeep tail

p50: 35ms — cache hit
p90: 80ms — cache miss
p95: 150ms — miss + contention
p99: 900ms — miss + DB wait
p99.9: 4,000ms — + GC pause

Command

text

pN = the latency value at or below which N% of
     requests fall, after sorting all latencies
     fastest to slowest

p50 = median · p99 = slowest 1% starts here

pN = the latency N% of requests fall at or below. p50 = median; p99/p99.9 expose the tail an average hides.

3.Why averages hide tail latency problems

An average is easily dominated by the common, fast requests — a small number of very slow outliers barely move it, even though every one of those outliers is a real user having a bad experience.

One sample, five statistics — 990 requests at 20ms, 10 at 5,000ms

Computed from the stated sample, not measured. mean = (990 x 20 + 10 x 5,000) / 1,000 = 69.8ms. p99 is the 990th value in sorted order, still 20ms; p99.9 is the 999th, 5,000ms. Four bars are almost invisible beside the fifth, and that is the point — every statistic up to p99 reports a healthy service while ten users wait five seconds.

Command

text

mean = sum(all latencies) / count

a few very slow requests barely move the mean,
because they're divided across a much larger count
of fast ones — the mean dilutes outliers

A mean dilutes a few very slow requests across many fast ones — it can look healthy while a real tail of bad experiences exists underneath it.

4.Little's Law: concurrency ≈ throughput × latency

Little's Law connects the two quantities this section opened with: the number of requests being handled at once (concurrency) equals throughput multiplied by latency. Raise either throughput or latency and required concurrency rises with it.

Key features

Concurrency ≈ throughput × latency (average number in the system at once)
Throughput = requests/sec; latency = seconds/request; concurrency = unitless count
Raising throughput without adding capacity raises required concurrency
Raising latency (a slower downstream call) also raises required concurrency, at the same throughput
Used to size worker pools, connection pools and thread pools from a target throughput and latency

Little's Law in practice

Little's Law in practice
Given	Formula	Worked example
Throughput + latency	concurrency = throughput × latency	500 req/s × 0.2s = 100 concurrent requests
Concurrency + latency	throughput = concurrency ÷ latency	100 workers ÷ 0.2s = 500 req/s max
Concurrency + throughput	latency = concurrency ÷ throughput	100 ÷ 500 req/s = 0.2s average latency

A downstream slowdown caps throughput, at a fixed pool size

× latencyexceedscapacity

Target throughput

1,000 req/s

Latency rises

150ms -> 600ms

Concurrency needed

1,000 × 0.6 = 600

Fixed pool (150)

caps actual throughput at 250 req/s

Command

text

concurrency = throughput (req/s) × latency (s)

rearranged:
throughput = concurrency / latency
latency    = concurrency / throughput

concurrency ≈ throughput × latency — raising either raises how many requests must be in flight at once.

5.Serial vs parallel portions of a request path

A request often calls several downstream services. Steps that must happen one after another (serial) add their latencies together; steps that can run at the same time (parallel) only cost the slowest one. Total latency depends on which shape the path actually has.

Serial vs parallel cost

Serial vs parallel cost
Shape	Total latency	Example
Serial (A then B)	latency(A) + latency(B)	authenticate, then fetch the authorized resource
Parallel (A and B)	max(latency(A), latency(B))	fetch profile and fetch orders at the same time

Same four calls, two shapes

Bars are drawn to scale in milliseconds. Only the auth call has a real dependency; the code order created the other three.

Command

text

serial:   total = sum(step latencies)
parallel: total = max(step latencies)

a request path is usually a mix of both

Serial steps sum their latencies; independent steps run in parallel and only cost the slowest one. Identify real dependencies, not code order.

Spotted a mistake on this page? Tell us.

Read Sheet