beginnerintermediateadvanced

System Design · Section 32

Timeouts and Resource Limits

Level: intermediate
Read: 20 min
Concepts: 3

A network call needs three distinct deadlines, not one: a connect timeout for reaching the dependency at all, a read timeout for the wait between chunks once connected, and a request timeout for the entire call end to end — a read-only timeout cannot bound a response that keeps trickling data just under its threshold forever. Beyond timeouts, six further resources each need an explicit cap or they default to unbounded: queue depth, request/upload body size, concurrency, database connections, and memory. Finally, every downstream timeout in a call chain has to fit within the caller's own deadline — sequential calls consume that budget additively while parallel calls to independent dependencies consume only the maximum of the branch, which is why deadline propagation (passing the actual remaining time down the chain) beats assigning each hop a fixed, independently-chosen timeout.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Connect, read and request timeouts bound three genuinely different failure modes — a service needs all three, not just one.
Queue depth, body/upload size, concurrency, database connections and memory each need their own explicit cap beyond any timeout.
A caller's deadline is a shared, finite budget every downstream timeout is carved out of — sequential calls sum, parallel calls take the max.
Deadline propagation (recomputing the remaining budget at each hop) avoids downstream timeouts that individually look reasonable but sum to more than the caller can actually wait.

What you will be able to do

Configure connect, read and request timeouts correctly for a given network dependency
Identify which of the six resource limits a given service is missing before it becomes an incident
Compose timeout budgets correctly across a multi-hop call chain, using deadline propagation rather than fixed independent values

Bounding a single call and its resources

The three deadlines a network call needs, and the further resource limits a timeout alone does not cover.

Connect, read and request deadlines are three different timeouts

coreintermediate

A single "timeout" setting is not enough for a network call — three distinct deadlines cover three distinct failure modes. A connect timeout bounds how long to wait for the TCP handshake/connection to be established at all. A read timeout bounds how long to wait for data once the connection is open (a server that accepted the connection but never responds). A request (or total) timeout bounds the entire call end to end, including retries or streaming reads that could otherwise continue indefinitely one chunk at a time. Setting only one of these leaves the other failure modes completely unbounded.

Think of it as

Ordering food at a restaurant has the same three deadlines. The connect timeout is how long you wait for someone to even come take your order — if nobody shows up in five minutes, you leave, no food was ever discussed. The read timeout is how long you wait between the waiter saying "I'll check on that" and them actually coming back — if they vanish for 40 minutes mid-conversation, something is wrong even though the "connection" (your table) is still technically open. The request timeout is a hard rule for the whole meal — even if the waiter keeps coming back with small updates every few minutes forever, you still have to leave for your next appointment at some point regardless of how the individual check-ins went.

python

import requests

requests.get(
    "https://payments.internal/charge",
    timeout=(3, 10),  # (connect_timeout, read_timeout) in seconds
)

What we're doing: Show a read-timeout-only configuration failing to bound a slow-trickle response that a request timeout would have caught.

read-timeout-gap.txttext

Client config: connect_timeout = 3s, read_timeout = 5s
                (no overall request timeout set)

A dependency streams its response one small chunk
every 4 seconds, forever, without ever completing.

t=0s   connection established (under 3s connect timeout)
t=4s   first chunk arrives (under 5s read timeout, resets it)
t=8s   second chunk arrives (under 5s read timeout, resets it)
t=12s  third chunk arrives (under 5s read timeout, resets it)
...    this continues indefinitely — each individual gap
       between chunks is under the read timeout, so the
       read timeout never fires, and the call never ends.

With a request timeout of, say, 15s added: the call is
forcibly aborted at t=15s regardless of how the individual
read gaps looked, because the TOTAL time is what's bounded.

9: Each individual chunk arrives well within the 5-second read timeout, so from the read timeout's perspective nothing is ever wrong.
12: This is the gap: a read timeout only measures the wait between chunks, never the call's total duration — a request timeout is the only deadline that bounds this.

Why this works: This is the concrete reason a read timeout is not a substitute for a request timeout — a dependency does not need to go fully silent to hang a caller indefinitely, it only needs to keep the gaps between chunks just under the read-timeout threshold.

Setting only a read timeout and assuming it bounds the whole call

Wrong

text

requests.get(url, timeout=5)
# a single number in most client libraries maps
# to the READ timeout only, not connect or total

Better

text

requests.get(url, timeout=(3, 5))
# explicit (connect, read) tuple, PLUS a
# request-level deadline enforced by the caller
# (e.g. a surrounding cancellation context or
# an explicit deadline check) for the total call

What you see: A request that should have failed fast instead hangs for minutes, consuming a thread or connection slot the whole time, while every individual timeout value in the configuration looks reasonable in isolation — the gap is that no single configured value actually bounds the total call duration.

Why: Many HTTP client libraries default a bare `timeout=N` parameter to only the read timeout (or a combined connect+read that still does not bound streaming/chunked responses) — assuming it covers the whole request leaves exactly the slow-trickle failure mode from the example above completely unbounded.

Three distinct deadlines

Connect timeout

time to establish the connection

Read timeout

time between chunks

Request timeout

bounds the entire call

The three deadlines and what each one alone catches

The three deadlines and what each one alone catches
Deadline	Bounds	Catches
Connect timeout	Time to establish the connection	An unreachable or overloaded dependency that never accepts the connection
Read timeout	Time between chunks of data once connected	A connected dependency that stalls mid-response
Request timeout	The entire call, start to finish	A dependency that keeps trickling data forever without completing

Remember: A connect timeout bounds reaching the dependency at all; a read timeout bounds the wait between chunks once connected; a request timeout bounds the entire call. Setting only one leaves the other two failure modes completely unbounded — a chunked response trickling data just under the read timeout can hang forever without a request timeout to catch it.

The resource-limit checklist: queue depth, body size, concurrency, connections, memory

coreintermediate

A timeout alone does not stop a service from being overwhelmed — it only bounds how long any single request waits. A service also needs explicit caps on every other resource an unbounded or malicious caller could exhaust: how many requests can queue up waiting to be processed, how large a single request body or upload can be, how many requests can run concurrently, how many database connections the service can hold open at once, and how much memory any single operation is allowed to consume. Without an explicit limit, each of these defaults to "however much the caller sends," which means an unbounded input directly becomes an unbounded resource commitment.

Think of it as

Think of a small coffee shop with no policies at all beyond "we serve everyone." Without a queue cap, the line stretches out the door and around the block, and the people at the back wait so long they might as well not be in line. Without an order-size cap, one customer can order 500 drinks in a single order and tie up the espresso machine for an hour. Without a concurrency cap, every barista tries to make every drink at once and nothing actually finishes efficiently. Without a limit on how many suppliers' trucks (database connections) can be unloading in the shop's one loading dock at a time, deliveries jam the dock and block each other. Every one of these is a different resource, and a shop that only says "please don't take too long" (a timeout) has said nothing about any of them.

yaml

# example service resource-limit config
queue_max_depth: 500
request_body_max_bytes: 10_485_760      # 10 MB
upload_max_bytes: 104_857_600            # 100 MB
concurrency_max: 200
db_pool_max_connections: 20
request_memory_max_bytes: 268_435_456    # 256 MB

What we're doing: Show an unbounded request body size turning a single request into a memory exhaustion incident.

unbounded-body-size.txttext

Service: image upload endpoint, no max body size
        configured, reads the entire request body
        into memory before validating it.

Normal traffic: uploads average 2MB, well within
what the service was tested with.

Incident: a single request arrives with a 40GB
request body (either a mistake, or a deliberate
denial-of-service attempt).

The service reads the ENTIRE body into memory
before it ever gets to check the file size or
reject it — by the time validation would have
run, the process has already tried to allocate
40GB, triggering an out-of-memory kill that takes
down every OTHER request being served by that
same process, not just the one bad upload.

9: This is the design flaw: validation happens AFTER the body is fully read into memory, so a size check that exists in the code never gets a chance to run before the damage is done.
13: The blast radius extends well past the one bad request — every other request sharing that process goes down with it.

Why this works: This is the concrete failure a body-size limit exists to prevent — the limit has to be enforced at the transport/streaming layer, before or during the read, not as a post-hoc check on an already-fully-buffered payload.

Checking a size limit only after reading the entire body into memory

Wrong

python

body = request.read()          # reads everything,
                                 # regardless of size
if len(body) > MAX_SIZE:
    return error(413)

Better

python

content_length = request.headers.get('Content-Length')
if content_length and int(content_length) > MAX_SIZE:
    return error(413)          # rejected before
                                 # reading the body

# and/or stream-read with an enforced cap:
body = request.read(max_bytes=MAX_SIZE)

What you see: A memory limit configured in application code never actually prevents an out-of-memory incident, because the code path that checks it runs strictly after the code path that already allocated the memory it was supposed to prevent allocating.

Why: A size check on an in-memory variable can only ever fire after that variable already exists in memory at full size — real protection means rejecting or streaming based on the declared or observed size before or during the read, not validating a value that has already cost the resource it was meant to bound.

Five resource limits, each defaults to unbounded

Queue depth

max waiting requests

Body/upload size

max payload bytes

Concurrency

max simultaneous work

DB connections

bounded pool size

Memory

per-request cap

Six resource limits and what happens without each one

Six resource limits and what happens without each one
Resource	Without a limit	Typical mechanism
Queue depth	Requests wait indefinitely, doing wasted work once finally processed	Bounded queue, reject with 503 once full
Request/upload body size	One request can consume unbounded bandwidth/memory	Max content-length enforced before or during read
Concurrency	Unlimited simultaneous work overwhelms CPU and downstream dependencies	Semaphore, worker pool size, connection limits
Database connections	One service can exhaust the database's total connection limit	Bounded connection pool per instance
Memory	A single pathological input can trigger an OOM affecting all requests	Per-request memory caps, streaming instead of full buffering

Remember: A timeout alone only bounds how long one request waits — queue depth, request/upload body size, concurrency, database connections and memory each need their own explicit cap, or they default to "however much the caller sends," turning any one of them into an unbounded resource commitment.

Composing a call chain's timeout budget

Why individual downstream timeouts must be derived from the caller's remaining deadline, not chosen independently.

Timeout budgets must fit within the overall request latency budget

coreintermediate

A single incoming request often triggers a chain of downstream calls, each with its own timeout. If those individual timeouts are not deliberately chosen to sum to less than the caller's own deadline, the caller can time out and give up while its downstream calls are still running — wasting the work those calls were doing and giving the end user a failure even though the system was technically still making progress. A timeout budget means treating the overall request deadline as a fixed pool that every downstream timeout has to be carved out of, not choosing each downstream timeout independently as if it had the whole budget to itself.

Think of it as

A request's total time budget is like a fixed amount of cash for a road trip with several toll stops. If you do not plan ahead and each toll booth assumes it can take "whatever it needs" from your wallet, you can run out of money at the third booth even though each individual toll, in isolation, seemed completely reasonable. The fix is deciding upfront how much of the total trip budget each leg is allowed to spend, so the sum across all legs never exceeds what you actually brought — not hoping it works out because no single toll looked expensive on its own.

text

caller_deadline = 8s (fixed, set once at the top)

hop 1 starts at t=0, remaining budget = 8s
  -> hop 1's own timeout should be < 8s, leaving
     room for hops 2 and 3

hop 1 finishes at t=3s, remaining budget = 8-3 = 5s
hop 2 starts, its timeout should be < 5s (not a
  fixed 5s independent of what's already elapsed)

What we're doing: Show a request handler whose downstream timeouts sum to more than its own caller deadline, and the fix using deadline propagation.

timeout-budget.pypython

# BROKEN: each downstream call gets a fixed 5s
# timeout, independent of the caller's own deadline
# or how much time earlier calls already used.
def handle_request():
    # caller's own deadline: 8 seconds total
    user = fetch_user(timeout=5)        # up to 5s
    orders = fetch_orders(timeout=5)    # up to 5s more
    # worst case: 10s, already past the 8s deadline
    # the caller was actually given
    return combine(user, orders)

# FIXED: each call's timeout is carved out of the
# ACTUAL remaining budget, not a fixed independent value.
def handle_request_budgeted():
    deadline = now() + 8  # the real caller deadline
    remaining = deadline - now()
    user = fetch_user(timeout=remaining)
    remaining = deadline - now()   # recompute after
                                     # the first call
    orders = fetch_orders(timeout=remaining)
    return combine(user, orders)

8: This is the bug: 5s + 5s can exceed the caller's real 8-second deadline, but neither individual timeout value looks wrong on its own.
17: Recomputing the remaining budget after each call is what deadline propagation actually means — each hop gets what is truly left, not a value chosen in isolation.

Why this works: This is the exact mechanism by which individually-reasonable timeout values combine into an unreasonable total — the fix is not choosing smaller individual timeouts by guesswork, it is deriving each one from the actual remaining budget at the moment that call starts.

Choosing every downstream timeout independently without checking they sum to less than the caller's deadline

Wrong

text

"Each of our 4 downstream calls has a
generous 3-second timeout — plenty of room."
# 4 x 3s = 12s worst case, for a caller whose
# own upstream deadline is 8 seconds

Better

text

"Our caller deadline is 8 seconds. 4 sequential
calls need to share that budget — either run
them in parallel (worst case 3s, not 12s), or
give each one a fraction of the remaining budget
computed at call time, not a fixed independent
value."

What you see: A request fails with a timeout at the top level, but every individual downstream service's own logs and dashboards show its calls completing successfully and quickly — because each downstream service really was fast in isolation, the caller had simply already given up by the time all of them finished in sequence.

Why: Choosing each timeout to look reasonable on its own is not the same as choosing timeouts that compose correctly across a call chain — the caller's real deadline is a shared, finite budget, and a design that never checks whether individual timeouts sum to less than it will eventually produce a request chain that individually "worked" everywhere but still failed overall.

3 calls, 5s each, against an 8s caller deadline

Sequential

+Worst case: 15s (sum)
+Blows past the 8s caller deadline
+Each timeout looked fine in isolation

Parallel

−Worst case: 5s (max of the branch)
−Fits comfortably within the 8s deadline
−Only valid when the calls are truly independent

Sequential vs parallel calls against the same total budget

Sequential vs parallel calls against the same total budget
Call pattern	Worst-case total latency	Budget implication
3 sequential calls, 5s timeout each	15s (sum)	Needs a 15s+ caller budget to ever complete successfully
3 parallel calls, 5s timeout each	5s (max)	Fits comfortably within an 8s caller budget
3 sequential calls, budget-aware (deadline propagation)	Bounded by the caller's own remaining deadline	Each hop gets exactly what is left, never more than the caller actually has

Remember: A caller's deadline is a shared, finite budget every downstream timeout is carved out of. Sequential calls consume it additively (sum); parallel calls to independent dependencies consume it as the max of the branch. Recompute the remaining budget at each hop (deadline propagation) rather than assigning each downstream timeout a fixed, independently-chosen value.