beginnerintermediateadvanced

AWS · Performance

Performance and Scalability

The scaling levers, measuring the bottleneck, the vocabulary, and validating by test

Three kinds of leveradd capacity, remove work, remove contention
Scaling out can make it worsewhen the constraint is shared and bounded
Tail amplifies with fan-out20 calls at p99 100 ms ≠ a fast page
Test with production data volumea small dataset makes every query fast

1.The Scaling Toolbox

There are about ten ways to make a system handle more load, and they are not interchangeable. Some add capacity (scale out, scale up, autoscale). Some remove work (cache, CDN, batch). Some move work off the critical path (async processing, queues). Some remove contention (pooling, partitioning, workload isolation). The measurement tells you which category you need.

Key features

AWS defines performance efficiency as using computing resources efficiently to meet system requirements, and maintaining that efficiency as demand changes and technologies evolve
Horizontal scaling adds instances and needs statelessness; vertical scaling changes instance size and usually needs a restart, so it does not help availability
Caching and CDN remove work rather than adding capacity, which is why they are usually the cheapest large win — and why a low hit rate makes them worthless
Async processing and batching move work off the request path; the user-visible latency drops even though total work does not
Connection pooling addresses contention, not throughput — a database with a bounded connection count is a shared resource that pooling rations
Read replicas scale reads only and add replication lag, partitioning scales writes but changes the data model, and workload isolation stops a batch job degrading interactive traffic

Which lever for which bottleneck

Which lever for which bottleneck
Bottleneck	Lever	Why the obvious one fails
CPU-bound application tier	Scale out	A bigger instance helps once; more instances keep helping
Repeated identical reads	Cache or CDN	More capacity pays for the same work over and over
Slow third-party call in the request path	Async processing	Scaling out multiplies the waiting, not the throughput
Database connections exhausted	Pooling, then a proxy	More application instances make this strictly worse
Read-heavy database	Read replicas	Watch replication lag before routing reads
Write-heavy database	Partitioning	Replicas do not help writes at all
Batch job starving interactive traffic	Workload isolation	Autoscaling hides it by paying for both

Three kinds of lever

Add capacity

Scale out

more instances; needs statelessness

Scale up

bigger instance; usually a restart

Autoscale

track a signal users feel

Remove work

Cache

worthless at a low hit rate

CDN

serve static bytes at the edge

Batch

amortize per-call overhead

Remove contention

Connection pooling

ration a bounded resource

Partitioning

scales writes, changes the model

Workload isolation

batch must not degrade interactive

Signature

add capacity (out/up/auto) · remove work (cache, CDN, batch, async) · remove contention (pool, partition, isolate)

Three categories: add capacity (scale out/up, autoscale), remove work (cache, CDN, batch), remove contention (pooling, partitioning, isolation). Measure which one you have before choosing — scaling out a contention bottleneck makes it worse, not better.

2.Measure the Bottleneck Before Scaling

Scaling without measuring is buying capacity for a constraint you have not identified. The measurement is a breakdown, not a number: of the total time a request takes, how much is your code, how much is waiting on a database, how much is waiting on another service, and how much is waiting for a resource to become free.

Key features

Utilization plus queueing is the signal: a resource at 100% with work waiting is the bottleneck, one at 100% with nothing waiting is merely well used
A trace gives the breakdown directly — time in your code, in the database, in downstream calls, and in the gap that is network and connection acquisition
Saturation metrics beat utilization metrics for shared resources: connection pool wait time says more than database CPU
Removing one bottleneck reveals the next, so measurement is a loop rather than a step
Scaling before measuring changes the system, which destroys the baseline you would have compared against
Cost is part of the measurement: two fixes that both work are not equivalent if one costs ten times more per month

Where to look, in order

Where to look, in order
Question	Where the answer is	What it rules out
Is it slow for everyone, or only some requests?	Latency percentiles by endpoint	A general capacity problem, if only one endpoint moved
Where does the time go inside a request?	A trace with subsegments	Everything except the segment that dominates
Is a resource saturated?	Pool wait time, queue depth, concurrency vs limit	Capacity, when utilization is low but waiting is high
Did it change, or has it always been like this?	The same metric over weeks	A recent regression, or confirms one
Does load correlate with the slowdown?	Latency plotted against throughput	Load as the cause, if latency is flat against it

Signature

breakdown = own code + downstream + database + resource wait · bottleneck = 100% utilized with work queued

Measure the breakdown before spending: own code, downstream, database, and time waiting for a resource. Utilization plus queueing identifies the bottleneck; percentiles and saturation metrics are what make it visible. Then fix the dominant term, and measure again.

3.Throughput, Latency, Concurrency, and Tail Latency

Six words carry most performance conversations. Latency is how long one request takes. Throughput is how many complete per second. Concurrency is how many are in flight at once. Queue depth is how many are waiting. Saturation is how close a resource is to its limit. Tail latency is what the unluckiest requests experience — and it is what users remember.

Key features

Concurrency ≈ throughput × latency (Little's law): halving latency at constant concurrency doubles throughput
Throughput and latency move together only until saturation; past it, throughput plateaus and latency rises steeply
Queue depth is the earliest visible symptom of saturation, because work waits before it fails
Tail latency dominates user experience in a system where one page makes many calls — the slowest of twenty calls sets the page time
Averages are a poor summary of a long-tailed distribution: p50 can improve while p99 gets worse, and users feel the p99
A retry converts one slow request into two, so retries increase both throughput demand and tail latency exactly when the system is struggling

The six terms, precisely

The six terms, precisely
Term	Unit	What it tells you
Latency	Milliseconds, at a percentile	How long one request takes
Throughput	Requests per second	How much work completes
Concurrency	In-flight requests	How much is happening at once (≈ throughput × latency)
Queue depth	Items waiting	How far demand exceeds capacity, right now
Saturation	Percent of a hard limit	How close a resource is to refusing work
Tail latency	p99, p99.9	What the unluckiest users experience

What happens as load rises

Below saturation

throughput rises, latency flat

Approaching saturation

queue depth starts to grow

At saturation

throughput plateaus, latency climbs

Past it

timeouts, retries, collapse

Signature

concurrency ≈ throughput × latency · saturation → queue depth → latency → timeouts · tail amplifies with fan-out

concurrency ≈ throughput × latency. Throughput plateaus at saturation while latency climbs, and queue depth grows first. Tail latency amplifies across fan-out — twenty calls at p99 = 100 ms give a much worse page p99 — so report p50, p95, and p99 together.

4.Load Testing and Validating Scaling Assumptions

Every scaling design contains assumptions: that the service scales linearly, that the database can take the extra connections, that autoscaling reacts fast enough. A load test turns those assumptions into measurements before traffic does. Production metrics then confirm the test resembled reality.

Key features

AWS recommends load testing to identify which quotas will limit the traffic levels you expect, and to act on them before deploying
Test in an environment with production-equivalent quotas and data volume, or the results describe a system you do not run
Autoscaling reaction time is part of the result: a step change in load reveals whether the policy scales fast enough or whether the queue absorbs the gap
Cache behaviour differs enormously between a test with 100 items and production with 10 million — hit rate is the variable that changes most
Ramp to find the knee (where latency starts climbing), then hold there to find what breaks first — the two answers are different
Production metrics validate the test: if the real traffic mix, payload sizes, or cache hit rate differ, the test's conclusions do not transfer

What each kind of test tells you

What each kind of test tells you
Test	Shape	Answers
Ramp / capacity test	Increase load until latency climbs	Where the knee is, and which resource saturates first
Soak test	Hold moderate load for hours	Leaks, connection exhaustion, log and disk growth
Spike test	Step change in load	Whether autoscaling and queues absorb a sudden burst
Failure injection	Remove a dependency under load	Whether timeouts and degradation behave as designed
Production canary	A small share of real traffic	Whether the test's assumptions held at all

Command

text

# A load test that is worth trusting
1. Production-equivalent quotas, instance sizes, and data volume
2. The real endpoint mix and payload sizes, not one hot path
3. Ramp to find the knee; hold to find what breaks
4. Record which resource saturated, not just the rps number
5. Compare the traffic profile against production afterwards

Test with production-equivalent quotas and data volume, using the real endpoint mix. Ramp to find the knee, hold to find what breaks, and record which resource saturated rather than only the requests-per-second number. Then check the real traffic against the test's assumptions.

Spotted a mistake on this page? Tell us.

Read Sheet