beginnerintermediateadvanced

System Design · Timeouts and Resource Limits

Timeouts and Resource Limits

Three deadlines, six resource caps, and why they have to compose across a call chain

Three deadlinesconnect, read, and total request
Six limitsqueue depth, body size, concurrency, connections, memory
Budget compositionsequential sums, parallel takes the max
Deadline propagationpass remaining time, not a fixed value

1.Connect, read and request deadlines are three different timeouts

A single "timeout" setting is not enough for a network call — three distinct deadlines cover three distinct failure modes. A connect timeout bounds how long to wait for the TCP handshake/connection to be established at all. A read timeout bounds how long to wait for data once the connection is open (a server that accepted the connection but never responds). A request (or total) timeout bounds the entire call end to end, including retries or streaming reads that could otherwise continue indefinitely one chunk at a time. Setting only one of these leaves the other failure modes completely unbounded.

Key features

Connect timeout: bounds TCP handshake / connection establishment — protects against a dependency that is unreachable, not just slow to respond
Read timeout (also called socket timeout): bounds the wait for data once connected — protects against a dependency that accepted the connection but is hung or silently stalled
Request timeout (total/overall timeout): bounds the entire operation end to end, including any streaming or chunked reads — protects against a dependency that keeps sending small amounts of data forever without actually finishing
A read timeout alone does not bound a chunked/streaming response that keeps arriving one byte at a time just under the read-timeout threshold — this is exactly what a request timeout catches and a read timeout cannot
Most HTTP client libraries (e.g. Python requests, Go net/http, Java HttpClient) expose these as separate configurable values precisely because a single combined timeout cannot express all three failure modes correctly

The three deadlines and what each one alone catches

The three deadlines and what each one alone catches
Deadline	Bounds	Catches
Connect timeout	Time to establish the connection	An unreachable or overloaded dependency that never accepts the connection
Read timeout	Time between chunks of data once connected	A connected dependency that stalls mid-response
Request timeout	The entire call, start to finish	A dependency that keeps trickling data forever without completing

Three distinct deadlines

Connect timeout

time to establish the connection

Read timeout

time between chunks

Request timeout

bounds the entire call

Command

python

import requests

requests.get(
    "https://payments.internal/charge",
    timeout=(3, 10),  # (connect_timeout, read_timeout) in seconds
)

A connect timeout bounds reaching the dependency at all; a read timeout bounds the wait between chunks once connected; a request timeout bounds the entire call. Setting only one leaves the other two failure modes completely unbounded — a chunked response trickling data just under the read timeout can hang forever without a request timeout to catch it.

2.The resource-limit checklist: queue depth, body size, concurrency, connections, memory

A timeout alone does not stop a service from being overwhelmed — it only bounds how long any single request waits. A service also needs explicit caps on every other resource an unbounded or malicious caller could exhaust: how many requests can queue up waiting to be processed, how large a single request body or upload can be, how many requests can run concurrently, how many database connections the service can hold open at once, and how much memory any single operation is allowed to consume. Without an explicit limit, each of these defaults to "however much the caller sends," which means an unbounded input directly becomes an unbounded resource commitment.

Key features

Queue depth: the maximum number of requests allowed to wait before being rejected outright — an unbounded queue means requests can wait so long that by the time they're processed, the caller has already given up or timed out, wasting the work entirely
Request body size / upload size: an explicit maximum on how large a single request payload can be — without one, a single request can consume unbounded bandwidth and memory just parsing or buffering it
Concurrency: a cap on how many requests (or how many instances of an expensive operation) run at the same time — protects CPU, memory and downstream dependencies from being hit by unlimited simultaneous work
Database connections: a bounded connection pool per service instance — an unbounded number of connections can exhaust the database server's own connection limit, starving every other service sharing that database
Memory: explicit limits on how much memory a single request or background job may allocate — without one, a single pathological input (a very large file, a deeply nested payload) can trigger an out-of-memory condition that affects every other request on the same process

Six resource limits and what happens without each one

Six resource limits and what happens without each one
Resource	Without a limit	Typical mechanism
Queue depth	Requests wait indefinitely, doing wasted work once finally processed	Bounded queue, reject with 503 once full
Request/upload body size	One request can consume unbounded bandwidth/memory	Max content-length enforced before or during read
Concurrency	Unlimited simultaneous work overwhelms CPU and downstream dependencies	Semaphore, worker pool size, connection limits
Database connections	One service can exhaust the database's total connection limit	Bounded connection pool per instance
Memory	A single pathological input can trigger an OOM affecting all requests	Per-request memory caps, streaming instead of full buffering

Five resource limits, each defaults to unbounded

Queue depth

max waiting requests

Body/upload size

max payload bytes

Concurrency

max simultaneous work

DB connections

bounded pool size

Memory

per-request cap

Command

yaml

# example service resource-limit config
queue_max_depth: 500
request_body_max_bytes: 10_485_760      # 10 MB
upload_max_bytes: 104_857_600            # 100 MB
concurrency_max: 200
db_pool_max_connections: 20
request_memory_max_bytes: 268_435_456    # 256 MB

A timeout alone only bounds how long one request waits — queue depth, request/upload body size, concurrency, database connections and memory each need their own explicit cap, or they default to "however much the caller sends," turning any one of them into an unbounded resource commitment.

3.Timeout budgets must fit within the overall request latency budget

A single incoming request often triggers a chain of downstream calls, each with its own timeout. If those individual timeouts are not deliberately chosen to sum to less than the caller's own deadline, the caller can time out and give up while its downstream calls are still running — wasting the work those calls were doing and giving the end user a failure even though the system was technically still making progress. A timeout budget means treating the overall request deadline as a fixed pool that every downstream timeout has to be carved out of, not choosing each downstream timeout independently as if it had the whole budget to itself.

Key features

A request handler that calls three downstream services sequentially, each with its own 5-second timeout, has an effective worst-case latency of 15 seconds for that chain alone — even if the caller's own deadline is only 8 seconds
Deadline propagation: passing the REMAINING time budget down the call chain (not a fixed value at every hop) lets each downstream call know exactly how much time is actually left, rather than each layer guessing independently
Sequential calls consume the budget additively; parallel/concurrent calls to independent dependencies consume it as the maximum of the parallel branch, not the sum — restructuring sequential calls into parallel ones (where the calls are truly independent) is a direct way to fit more work into the same budget
A downstream timeout longer than the remaining caller budget is functionally useless — the caller will have already given up and moved on before that downstream timeout could ever fire on its own
Some RPC frameworks and languages support deadline propagation directly (e.g. gRPC context deadlines, Go's context.WithTimeout) so each hop automatically inherits a shrinking remaining budget instead of a fixed value re-specified at every layer

Sequential vs parallel calls against the same total budget

Sequential vs parallel calls against the same total budget
Call pattern	Worst-case total latency	Budget implication
3 sequential calls, 5s timeout each	15s (sum)	Needs a 15s+ caller budget to ever complete successfully
3 parallel calls, 5s timeout each	5s (max)	Fits comfortably within an 8s caller budget
3 sequential calls, budget-aware (deadline propagation)	Bounded by the caller's own remaining deadline	Each hop gets exactly what is left, never more than the caller actually has

3 calls, 5s each, against an 8s caller deadline

Sequential

Worst case: 15s (sum)
Blows past the 8s caller deadline
Each timeout looked fine in isolation

Parallel

Worst case: 5s (max of the branch)
Fits comfortably within the 8s deadline
Only valid when the calls are truly independent

Command

text

caller_deadline = 8s (fixed, set once at the top)

hop 1 starts at t=0, remaining budget = 8s
  -> hop 1's own timeout should be < 8s, leaving
     room for hops 2 and 3

hop 1 finishes at t=3s, remaining budget = 8-3 = 5s
hop 2 starts, its timeout should be < 5s (not a
  fixed 5s independent of what's already elapsed)

A caller's deadline is a shared, finite budget every downstream timeout is carved out of. Sequential calls consume it additively (sum); parallel calls to independent dependencies consume it as the max of the branch. Recompute the remaining budget at each hop (deadline propagation) rather than assigning each downstream timeout a fixed, independently-chosen value.

Spotted a mistake on this page? Tell us.

Read Sheet