beginnerintermediateadvanced

AWS · Section 48

Performance and Scalability

Level: advanced
Read: 35 min
Concepts: 4

Scaling techniques fall into three categories and are not interchangeable: adding capacity (scale out, scale up, autoscale), removing work (cache, CDN, batch, async), and removing contention (connection pooling, partitioning, workload isolation). Choosing between them requires a measurement rather than an instinct, because the same symptom — "it is slow at peak" — has different correct answers depending on whether the constraint is capacity, contention, or a single slow operation, and scaling out a contention bottleneck actively makes it worse. The measurement is a breakdown of where a request spends its time, plus saturation metrics like connection-pool wait and queue depth that utilization graphs hide. The vocabulary matters too: concurrency is approximately throughput times latency, throughput plateaus at saturation while latency climbs, and tail latency amplifies across fan-out so that twenty calls to a service with a 100 ms p99 produce a page that is far worse than 100 ms. Finally, scaling designs contain assumptions, and load testing is how they become measurements — against production-equivalent quotas, data volume, and traffic mix, ramping to find the knee and holding to find what breaks, with production metrics confirming that the test resembled reality.

This section

"Boosting Performance and Scalability: AWS ElastiCache with RDS | Optimizing Database Queries"Technology Crunch

What is true here

Three lever categories: add capacity, remove work, remove contention — pick from the measured bottleneck.
Scaling out a bounded shared resource degrades latency rather than improving it.
Measure the breakdown and the saturation metrics; averages and utilization both hide the problem.
concurrency ≈ throughput × latency; tail latency amplifies with fan-out.
Load test with production-equivalent quotas and data, and record which resource saturated.

What you will be able to do

Classify a bottleneck as capacity, contention, or a single slow operation, and pick the matching lever
Produce a latency breakdown and read saturation metrics rather than utilization alone
Use Little's law to size a worker pool or connection pool from real percentiles
Explain why tail latency, not average latency, determines how a fan-out system feels
Design a load test whose conclusions transfer to production

From a symptom to a fix you can justify

Latency, throughput, concurrency, tail

Measure the bottleneck

Pick the right lever

Load test and validate

Performance and Scalability

The scaling levers and when each applies, measuring the bottleneck before spending, the vocabulary that makes the conversation precise, and validating assumptions by test.

The Scaling Toolbox

coreadvanced

There are about ten ways to make a system handle more load, and they are not interchangeable. Some add capacity (scale out, scale up, autoscale). Some remove work (cache, CDN, batch). Some move work off the critical path (async processing, queues). Some remove contention (pooling, partitioning, workload isolation). The measurement tells you which category you need.

Think of it as

A queue at a counter can be fixed by opening more counters, serving each person faster, or having fewer people need to queue at all. Every scaling technique is one of those three, and picking the wrong one adds cost without shortening the queue.

What we're doing: Pick the right lever for an application that slows down under load.

lever-choice.txttext

Symptom: p99 rises from 200 ms to 4 s at peak. CPU is 25%.

Wrong reflex: scale out from 6 to 20 tasks. Each new task opens more
database connections, the pool saturates sooner, and p99 gets worse.

Measurement: the database connection wait time is 3.7 s of the 4 s.
The bottleneck is contention on a bounded resource, not capacity.

Right lever: a connection proxy plus a smaller per-task pool, then
caching the three queries responsible for most of the connections.
p99 returns to 250 ms at 6 tasks, with no additional compute.

1: CPU at 25% already rules out the capacity category, which is where most scaling instincts start.
5: Adding instances to a contention bottleneck makes it worse — this is the most expensive common mistake in this section.
9: The fix costs less than the original state, because the wrong lever had been pulled repeatedly before anyone measured.

Why this works: The categories matter because pulling the wrong lever is not merely ineffective — for contention bottlenecks it actively harms. Knowing which of the three kinds you have is worth more than knowing all ten techniques.

Scaling out a service that shares a bounded downstream

Wrong

text

# Latency is high -> raise desired count from 6 to 20

Better

text

# Measure where the time goes first. If it is waiting on a shared
# resource, scaling out multiplies the contention.

What you see: Latency worsens after scaling out, and worsens further with each additional instance — the opposite of the expected relationship.

Why: Horizontal scaling assumes each instance brings its own capacity. When the constraint is a shared resource with a fixed limit — a connection pool, a licence, a rate-limited API — every new instance takes a smaller share of the same fixed amount, so throughput per instance falls faster than instance count rises.

Three kinds of lever

Add capacity

Scale out

more instances; needs statelessness

Scale up

bigger instance; usually a restart

Autoscale

track a signal users feel

Remove work

Cache

worthless at a low hit rate

CDN

serve static bytes at the edge

Batch

amortize per-call overhead

Remove contention

Connection pooling

ration a bounded resource

Partitioning

scales writes, changes the model

Workload isolation

batch must not degrade interactive

Which lever for which bottleneck

Which lever for which bottleneck
Bottleneck	Lever	Why the obvious one fails
CPU-bound application tier	Scale out	A bigger instance helps once; more instances keep helping
Repeated identical reads	Cache or CDN	More capacity pays for the same work over and over
Slow third-party call in the request path	Async processing	Scaling out multiplies the waiting, not the throughput
Database connections exhausted	Pooling, then a proxy	More application instances make this strictly worse
Read-heavy database	Read replicas	Watch replication lag before routing reads
Write-heavy database	Partitioning	Replicas do not help writes at all
Batch job starving interactive traffic	Workload isolation	Autoscaling hides it by paying for both

Together

text

# The order that usually costs least
1. Remove the work    (cache, CDN, fix the query, drop the N+1)
2. Move it off the path (async, queue, batch)
3. Remove contention  (pool, isolate, partition)
4. Add capacity       (scale out, then autoscale)

Remember: Three categories: add capacity (scale out/up, autoscale), remove work (cache, CDN, batch), remove contention (pooling, partitioning, isolation). Measure which one you have before choosing — scaling out a contention bottleneck makes it worse, not better.

Measure the Bottleneck Before Scaling

coreadvanced

Scaling without measuring is buying capacity for a constraint you have not identified. The measurement is a breakdown, not a number: of the total time a request takes, how much is your code, how much is waiting on a database, how much is waiting on another service, and how much is waiting for a resource to become free.

Think of it as

A bottleneck is the one resource at 100% while everything else has headroom. Find it before spending — otherwise you scale the parts that were never the constraint, and the constraint stays exactly where it was.

What we're doing: Show why the same symptom leads to three different fixes.

same-symptom.txttext

Symptom in all three cases: "checkout is slow at peak".

Case A: CPU 95% on every task, queue depth rising, pool wait near zero.
  -> Genuine capacity shortage. Scale out.

Case B: CPU 20%, pool wait 3.8 s, database CPU 30%.
  -> Contention on connections. Pooling or a proxy; scaling out harms.

Case C: CPU 20%, pool wait near zero, one query at 3.9 s.
  -> A single slow query. Fix the query; nothing else moves the number.

One symptom, three measurements, three unrelated fixes.

2: The only case where scaling out is correct — and it is the fix people apply in all three.
6: Distinguishable from case A only by the saturation metric, which is the one most teams do not collect.
10: Distinguishable from case B only by looking inside the database time. A trace shows this in one view.

Why this works: The symptom is identical and the correct action is not. Without the breakdown, the choice is a guess with a monthly bill attached, and two of the three guesses make the system worse or merely more expensive.

Watching averages instead of percentiles and saturation

Wrong

text

# Dashboard: average latency, average CPU

Better

text

# p50/p95/p99 latency per endpoint, plus saturation: pool wait, queue
# depth, concurrency against its limit

What you see: Every graph looks healthy while users complain, because the tail is a small fraction of requests and the constrained resource is not one of the two being watched.

Why: Averages hide the distribution and utilization hides queueing. A resource can sit at moderate utilization while requests wait in line for it, and that waiting is invisible unless the saturation metric is collected specifically.

Where to look, in order

Where to look, in order
Question	Where the answer is	What it rules out
Is it slow for everyone, or only some requests?	Latency percentiles by endpoint	A general capacity problem, if only one endpoint moved
Where does the time go inside a request?	A trace with subsegments	Everything except the segment that dominates
Is a resource saturated?	Pool wait time, queue depth, concurrency vs limit	Capacity, when utilization is low but waiting is high
Did it change, or has it always been like this?	The same metric over weeks	A recent regression, or confirms one
Does load correlate with the slowdown?	Latency plotted against throughput	Load as the cause, if latency is flat against it

Together

text

# The breakdown that decides the fix
total p99            4,200 ms
  own code             120 ms   -> not the problem
  downstream calls     270 ms   -> not the problem
  database             3,900 ms -> here
    of which query        40 ms
    of which pool wait 3,860 ms -> contention, not query speed

Remember: Measure the breakdown before spending: own code, downstream, database, and time waiting for a resource. Utilization plus queueing identifies the bottleneck; percentiles and saturation metrics are what make it visible. Then fix the dominant term, and measure again.

Throughput, Latency, Concurrency, and Tail Latency

coreintermediate

Six words carry most performance conversations. Latency is how long one request takes. Throughput is how many complete per second. Concurrency is how many are in flight at once. Queue depth is how many are waiting. Saturation is how close a resource is to its limit. Tail latency is what the unluckiest requests experience — and it is what users remember.

Think of it as

A supermarket checkout. Latency is one shopper's time in the queue and at the till. Throughput is shoppers per minute. Concurrency is how many tills are open. Queue depth is the line. Saturation is how busy the tills are. Tail latency is the shopper behind the person with a price check.

What we're doing: See why one page can be slow when every service behind it is fast.

tail-amplification.txttext

One service: p99 = 100 ms. That means 1 request in 100 takes 100 ms+.

A page that makes 20 independent calls to that service waits for the
slowest of the 20.

Probability all 20 are under the p99: 0.99^20 = 0.82.
So 18% of page loads contain at least one 100 ms+ call — the page's
p99 is far worse than any single service's p99.

This is why tail latency is a system property, not a service property.

1: Each service is meeting its target. Nobody in this system is doing anything wrong by their own dashboard.
5: Fan-out amplifies the tail. Reducing the number of calls per page is often more effective than making each call faster.

Why this works: Tail latency compounds across fan-out, which is why a microservice architecture can feel slow while every service reports healthy percentiles. The two levers are fewer calls per request and a tighter tail per call — not a lower average.

Sizing a pool from average latency instead of peak latency

Wrong

text

# 500 rps x 0.05 s average = 25 workers

Better

text

# 500 rps x 0.2 s p99 = 100 workers, so the pool survives the slow
# periods rather than only the typical ones

What you see: The pool is adequate most of the time and collapses during exactly the periods when latency rises — which is when capacity matters most.

Why: Little's law relates concurrency to the latency actually being experienced. When latency rises, required concurrency rises proportionally, so a pool sized from the average is undersized precisely during the slow periods that caused the problem.

What happens as load rises

Below saturation

throughput rises, latency flat

Approaching saturation

queue depth starts to grow

At saturation

throughput plateaus, latency climbs

Past it

timeouts, retries, collapse

The six terms, precisely

The six terms, precisely
Term	Unit	What it tells you
Latency	Milliseconds, at a percentile	How long one request takes
Throughput	Requests per second	How much work completes
Concurrency	In-flight requests	How much is happening at once (≈ throughput × latency)
Queue depth	Items waiting	How far demand exceeds capacity, right now
Saturation	Percent of a hard limit	How close a resource is to refusing work
Tail latency	p99, p99.9	What the unluckiest users experience

Together

text

# Little's law, used to size a worker pool
target throughput  = 500 requests/second
measured latency   = 0.2 seconds
required concurrency = 500 x 0.2 = 100 in-flight
# so 100 workers, or 100 connections, or 100 Lambda environments

Remember: concurrency ≈ throughput × latency. Throughput plateaus at saturation while latency climbs, and queue depth grows first. Tail latency amplifies across fan-out — twenty calls at p99 = 100 ms give a much worse page p99 — so report p50, p95, and p99 together.

Load Testing and Validating Scaling Assumptions

standardadvanced

Every scaling design contains assumptions: that the service scales linearly, that the database can take the extra connections, that autoscaling reacts fast enough. A load test turns those assumptions into measurements before traffic does. Production metrics then confirm the test resembled reality.

Think of it as

A load test is a rehearsal with a script you wrote. Its value depends entirely on whether the script matches what real users do — same mix of endpoints, same cache behaviour, same data volume. A test that only exercises the fastest path proves the fastest path is fast.

text

# A load test that is worth trusting
1. Production-equivalent quotas, instance sizes, and data volume
2. The real endpoint mix and payload sizes, not one hot path
3. Ramp to find the knee; hold to find what breaks
4. Record which resource saturated, not just the rps number
5. Compare the traffic profile against production afterwards

Load testing against a small dataset

Wrong

text

# Test database seeded with 5,000 rows; production has 40 million

Better

text

# Restore a recent production-sized snapshot (anonymized) into the test
# environment before measuring anything

What you see: The test shows a query at 4 ms and production runs the same query at 900 ms, because the small dataset fits entirely in memory and never needed the index that is missing.

Why: Query plans, cache hit rates, and index effectiveness are all functions of data volume. A small dataset makes almost every query fast, which means the test measures the framework overhead rather than the database — and misses exactly the problems that appear at scale.

What each kind of test tells you

What each kind of test tells you
Test	Shape	Answers
Ramp / capacity test	Increase load until latency climbs	Where the knee is, and which resource saturates first
Soak test	Hold moderate load for hours	Leaks, connection exhaustion, log and disk growth
Spike test	Step change in load	Whether autoscaling and queues absorb a sudden burst
Failure injection	Remove a dependency under load	Whether timeouts and degradation behave as designed
Production canary	A small share of real traffic	Whether the test's assumptions held at all

Together

text

# A ramp result worth acting on
 100 rps  p99  180 ms   pool wait   2 ms
 300 rps  p99  210 ms   pool wait   9 ms
 500 rps  p99  340 ms   pool wait  95 ms   <- the knee
 600 rps  p99 2,900 ms  pool wait 2.6 s    <- saturated
# Capacity is ~500 rps, and the binding resource is the connection pool

Remember: Test with production-equivalent quotas and data volume, using the real endpoint mix. Ramp to find the knee, hold to find what breaks, and record which resource saturated rather than only the requests-per-second number. Then check the real traffic against the test's assumptions.