beginnerintermediateadvanced

System Design · Section 67

Autoscaling

Level: intermediate
Read: 14 min
Concepts: 2

Autoscaling automates the horizontal pod/instance scaling decision covered in §13 (system-design.scaling.horizontal-vs-vertical) — a controller polls a metric and adds or removes instances when it crosses a target. The real design problem is which metric to poll: CPU and memory are cheapest to wire up but are only proxies for demand, queue depth is the right signal for worker/consumer tiers, request rate fits stateless web tiers, and custom metrics exist for whatever a workload's real bottleneck actually is. Because a new instance needs real time to cold-start (image pull, boot, warm-up, readiness checks) before it can serve traffic, autoscaling is reactive rather than instant — the full scaling lag stacks the polling interval, decision time, and cold start into one window where existing capacity absorbs a surge alone, which is what warm/pre-warmed standby capacity exists to hide. A noisy, unsmoothed metric makes this worse by triggering scale-up/scale-down thrashing for transient spikes that never reflected sustained load, each cycle still paying the full cold-start cost for no lasting benefit.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Autoscaling automates the §13 horizontal-scaling mechanism — the metric choice, not the add/remove-instance mechanism, is the design problem.
CPU/memory are cheapest but only proxies; queue depth and request rate track real bottlenecks more directly for worker and web tiers.
Cold start (image pull, boot, warm-up, readiness) means a new instance cannot help the instant it is requested.
Scaling lag stacks poll interval + decision time + cold start — existing instances absorb a surge alone for that whole window.
A noisy, unsmoothed metric causes scale-up/scale-down thrashing on transient spikes instead of reacting to sustained load.

What you will be able to do

Choose an autoscaling metric that matches a given workload's actual bottleneck rather than the easiest metric to read
Explain why autoscaling cannot prevent a request pile-up during a surge faster than its poll-interval-plus-cold-start window
Recognize scale-up/scale-down thrashing as a symptom of an unsmoothed, noisy metric rather than a capacity problem
Decide when warm/pre-warmed standby capacity is justified versus relying on reactive autoscaling alone

What drives the scaling decision

The metric choice behind horizontal pod/instance scaling — CPU, memory, request rate, queue depth, or a custom metric — and why the mechanism itself is §13 territory.

Autoscaling triggers and metrics

coreintermediate

Autoscaling is the automated version of the horizontal scaling decision covered in §13 (system-design.scaling.horizontal-vs-vertical) — instead of an engineer watching a dashboard and manually adding instances, a controller polls one or more metrics on a fixed interval and adds or removes instances when a metric crosses a threshold. The metric choice is the entire design problem: CPU and memory are the easiest to wire up (every host already reports them) but are only a proxy for the thing that actually matters — whether the service can keep up with demand. Request rate is closer to the real signal for a stateless web tier. Queue depth is closer still for a worker/consumer tier, because a worker can have low CPU while a queue backs up (the work is I/O-bound, or blocked waiting on a downstream dependency), which CPU-based scaling would completely miss. Custom metrics (e.g. in-flight requests per pod, p99 latency, a business metric like "carts pending checkout") exist because no built-in metric always reflects real load, and Kubernetes' Horizontal Pod Autoscaler explicitly supports external and custom metrics for exactly this reason.

Think of it as

Think of a restaurant deciding how many servers to have on shift. Scaling on CPU/memory alone is like deciding based on how tired the current servers look — a rough proxy, easy to observe, but disconnected from what customers actually experience. Scaling on request rate is like counting how many people walk in per minute. Scaling on queue depth is like counting how many parties are waiting for a table — the wait line is where a booked-solid but not-visibly-frantic restaurant reveals it is actually under strain. A custom metric is the manager's own judgment call, e.g. "the online order screen is backing up," when none of the generic counts capture the specific way this business gets overwhelmed.

yaml

# Kubernetes HPA scaling on a custom metric instead of raw CPU
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: queue_depth_per_worker
        target:
          type: AverageValue
          averageValue: "30"

What we're doing: Show why CPU-based scaling misses the bottleneck for a queue-consuming worker fleet.

worker-fleet-scaling.txttext

1. A worker fleet consumes jobs from a queue, calling
   a slow downstream payment API for each job.
2. The payment API gets slow. Workers now spend most
   of their time blocked waiting on that HTTP call.
3. CPU utilization per worker stays low -- the
   workers are idle-waiting, not computing.
4. A CPU-based autoscaler sees no pressure and does
   not add workers.
5. The queue depth climbs steadily because jobs
   arrive faster than the (still small) fleet can
   drain them, even though each worker looks "fine".
6. A queue-depth-based autoscaler would have added
   workers at step 5, which -- even blocked on the
   same slow API -- increases parallelism and drains
   the backlog faster.

9: This is the blind spot from the metrics table: CPU stays flat while the real bottleneck (queue backlog) is actively growing.
15: Queue depth reacts to the backlog directly, which is the signal that actually correlates with user-visible delay for this workload.

Why this works: CPU utilization is not wrong as a metric in general — it is wrong for this specific workload, where the bottleneck is I/O-bound waiting rather than compute. Picking a metric means picking one that actually tracks this workload's real constraint, not the easiest one to read.

Wiring the autoscaler to whichever metric is easiest to read, not the one that reflects the bottleneck

Wrong

yaml

# Worker fleet, scaled on CPU because it's the
# default metric every HPA example uses
metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Better

yaml

# Worker fleet, scaled on the metric that
# actually tracks this workload's bottleneck
metrics:
  - type: External
    external:
      metric:
        name: queue_depth_per_worker
      target:
        type: AverageValue
        averageValue: "30"

What you see: The queue backlog and end-to-end job latency both grow steadily during a downstream slowdown, while every dashboard showing CPU and memory looks calm — on-call has no autoscaling-related alert to look at because the metric it watches never left its target range.

Why: CPU utilization is the default example in almost every autoscaler tutorial, which makes it the path of least resistance regardless of whether it reflects the actual bottleneck for a given workload. A worker fleet blocked on downstream I/O is the textbook counter-example: the metric that is easiest to wire up is decoupled from the thing that is actually failing.

From metric to instance count

Metric polled

start

Compared to target

Replica count changed

end

Common autoscaling metrics and what they are actually proxies for

Common autoscaling metrics and what they are actually proxies for
Metric	What it directly measures	Best fit	Blind spot
CPU utilization	Processor busy time on the instance	Compute-bound stateless services	Misses I/O-bound or memory-bound bottlenecks entirely
Memory utilization	Working-set size on the instance	Memory-bound workloads (caches, in-memory aggregation)	Can stay flat while request latency is already degrading
Request rate	Incoming demand on a web/API tier	Stateless web tiers with roughly uniform per-request cost	Misleading if per-request cost varies wildly
Queue depth	Backlog waiting to be processed	Worker/consumer tiers, async pipelines	Reacts only after a backlog has already formed
Custom metric	Whatever the team defines (p99 latency, in-flight requests, a business counter)	Any workload whose real bottleneck no built-in metric captures	Only as good as the metric definition and its plumbing

Remember: Autoscaling automates the horizontal-scaling decision from §13 by polling a metric and comparing it to a target; the metric choice is the whole design problem, not an implementation detail. CPU/memory are cheapest to wire up but are only proxies — queue depth fits worker/consumer tiers, request rate fits stateless web tiers, and a custom metric exists for whatever workload-specific bottleneck no built-in metric captures. A configured autoscaler is not proof the right metric was chosen.

Why the decision is harder than it looks

Cold starts, warm capacity, scaling lag, and the danger of reacting to a noisy metric instead of sustained load.

Autoscaling failure modes: cold starts, warm capacity and scaling lag

coreintermediate

A new instance does not become useful the instant it is created — a cold start is the delay between "instance requested" and "instance actually serving traffic correctly": container image pull, process boot, JIT warm-up, cache population, and readiness-probe checks all take real time, sometimes tens of seconds. Warm capacity (pre-warmed or standby instances kept ready in advance) exists specifically to hide that delay from users, at the cost of paying for capacity that is idle most of the time. Scaling lag is the structural gap between a metric moving and new capacity actually absorbing load: the controller has to observe the metric (polling interval), decide to scale, and then wait out the cold start before the new instance helps at all — during that whole window, the existing instances are still absorbing the extra load alone. The fourth failure mode is a step further: if the metric driving the decision is noisy (a raw, un-smoothed value that spikes and drops from moment to moment rather than tracking real sustained load), the autoscaler reacts to noise instead of demand, causing scale-up/scale-down thrashing that wastes cold-start time on instances that were never actually needed.

Think of it as

Think of a coffee shop that only calls in an extra barista once the line is already out the door. Cold start is the time it takes that barista to arrive, clock in, and get up to speed — the line does not shrink the second they are called. Warm capacity is keeping one barista on standby in the back, paid to do nothing most of the day, purely so they can step in within seconds instead of minutes. Scaling lag is the whole stretch from "the line started growing" to "the new barista is actually taking orders" — existing staff absorb the surge alone the entire time. Scaling on a noisy metric is like calling in an extra barista every time three people walk in within the same ten seconds, even if the line clears itself moments later — by the time that barista arrives, the rush is already over, and they showed up for nothing.

yaml

# Kubernetes HPA stabilization window: dampen
# reaction to a noisy, spiky metric
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min of
                                        # sustained low load
                                        # before scaling down

What we're doing: Trace a traffic surge through the full scaling-lag window to show existing capacity absorbs it alone until cold start finishes.

scaling-lag-timeline.txttext

t=0s    Traffic doubles. Existing instances start
        queuing/slowing down.
t=15s   Autoscaler's next poll cycle observes the
        metric crossing its threshold.
t=16s   Scale-up decision made: +3 instances requested.
t=16-46s New instances pull images, boot, warm caches,
        and must pass readiness probes.
t=46s   New instances marked Ready and start receiving
        traffic.
-- Total lag: ~46 seconds where the original
   instances absorbed 2x load completely alone.

3: The polling interval alone accounts for up to one full cycle of pure delay before the autoscaler even notices.
6: Cold start (image pull + boot + warm-up + readiness) is usually the single largest component of the total lag, not the polling interval.
9: This is the scaling-lag window in full: existing capacity has no help for the entire 46 seconds, regardless of how correctly the autoscaler eventually reacted.

Why this works: Scaling lag is not one delay but a stack of several: detection delay (poll interval), decision delay, and cold start, all in sequence. Naming only "autoscaling" as the fix without accounting for this stacked delay understates how long a service is under-provisioned during a real surge.

Assuming autoscaling reacts fast enough to prevent a request pile-up during a sudden surge

Wrong

text

# Capacity planning note:
"We have autoscaling configured, so a
traffic spike will be handled automatically --
no need for standby capacity."

Better

text

# Capacity planning note:
"Autoscaling closes the gap after ~40-60s of
lag (poll interval + cold start). For surges
faster than that, we keep N warm/pre-warmed
instances on standby, or scale on a leading
indicator (queue depth) instead of a lagging
one (CPU) to start the cold start earlier."

What you see: A sudden, sharp traffic spike (a marketing push, a retry storm, a cache-invalidation stampede) causes a burst of timeouts and 5xx errors in the first 30-60 seconds, even though the autoscaler dashboard later shows it "correctly" scaled up and the incident quietly resolved itself — the postmortem finds the autoscaler behaved exactly as configured, and the actual problem was that nothing could have reacted fast enough.

Why: Autoscaling is reactive by construction — it can only add capacity after a metric has moved enough to be detected, and every new instance still has to pay the full cold-start cost before it helps. For any surge faster than (poll interval + cold start), the existing fleet must absorb the full spike alone regardless of how well-tuned the autoscaler is; the fix is warm capacity or a faster/leading metric, not a more aggressive autoscaler configuration.

The scaling-lag window during a traffic surge

Traffic surges

start

Metric crosses threshold

New instance boots

New instance serves traffic

end

The four failure modes and what actually causes each one

The four failure modes and what actually causes each one
Failure mode	Root cause	Mitigation
Cold start	New instance needs real time to boot, warm caches, and pass readiness checks before it can serve traffic	Smaller/faster images, pre-warmed pools, faster readiness checks
Warm capacity cost	Hiding cold start requires paying for standby instances that sit idle most of the time	Size the standby pool to expected surge frequency, not worst case
Scaling lag	Polling interval + decision time + cold start all elapse before new capacity helps at all	Scale on a leading indicator (queue depth, request rate) rather than a lagging one (CPU after the surge already hit)
Noisy-metric thrashing	Autoscaler reacts to short-lived spikes in an unsmoothed metric as if they were sustained load	Smoothing/averaging windows, stabilization windows, cooldown periods between scaling events

Remember: Cold start is the real time a new instance needs before it can serve traffic; warm capacity trades idle cost to hide that delay; scaling lag stacks poll interval + decision time + cold start into one window where existing capacity absorbs a surge alone. A noisy, unsmoothed metric makes all of this worse by triggering scale-up/scale-down cycles for transient spikes rather than sustained load — smoothing, stabilization windows, and cooldown periods exist specifically to tell the two apart.