beginnerintermediateadvanced

Python · Section 43

API Reliability

Level: advanced
Read: 160 min
Concepts: 9

Keeping a caller healthy when a downstream is not: bounding every call with a timeout, retrying only transient failures with exponential backoff and jitter, circuit breakers that fail fast once a downstream is known to be down, client-side rate limiting and idempotency as what makes a retry safe, request validation and input limits at the boundary, connection pooling and bulkheads as isolated finite resources, and graceful degradation via fallbacks when a dependency cannot be reached at all.

Python overview

Learn Python - Full Course for Beginners [Tutorial]freeCodeCamp.org

What is true here

Every network call needs an explicit timeout — the default in most Python HTTP clients is to wait forever.
Retry only transient failures, with exponential backoff and jitter — and only if the operation is idempotent or made idempotent with a dedup key.
A circuit breaker rejects calls immediately once a downstream is known to be failing, instead of paying the timeout cost on every single call.
A pool (connections, or a bulkhead's dedicated capacity) is finite by design — decide upfront what happens on exhaustion, not under load.
A fallback must catch only the specific failure it is meant to handle — a bare except in a fallback hides real bugs as if they were outages.

What you will be able to do

Set an explicit timeout on every network call, with separate connect and read budgets
Implement a retry loop with exponential backoff and jitter, and know which failures are safe to retry
Implement a circuit breaker with closed/open/half-open states, and explain when it is needed over retries alone
Explain the difference between server-side rate limiting and a client throttling its own calls to protect a downstream
Explain why idempotency is the precondition for a safe retry, not an optional add-on
Validate a request's shape and cap its size (string/list length, payload size) at the boundary
Explain a connection pool and a bulkhead as finite, isolated resources, and design for what happens when either is exhausted
Implement a fallback that degrades a response gracefully without masking a genuine bug as a downstream outage

Failing fast, not silently hanging

Bounding every call with a timeout, and retrying only transient failures with exponential backoff and jitter so a struggling downstream gets room to recover instead of being hit harder.

Timeouts

coreintermediate

A timeout is the maximum time you let a call to another service run before giving up and raising an error yourself. Without one, a hung downstream service hangs your service too — forever, not just slowly.

Think of it as

A timeout is a deadline you set, not one the other side agrees to. You are not asking the downstream service to hurry — you are deciding how long you personally are willing to wait before treating silence as a failure and moving on.

python

import httpx

with httpx.Client(timeout=5.0) as client:
    try:
        response = client.get(url)
    except httpx.TimeoutException:
        # decide here: retry, fall back, or surface an error
        ...

What we're doing: Set a short socket timeout against an address that will not respond, and confirm the call fails fast with socket.timeout instead of hanging.

connect_timeout.pypython

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(0.001)          # 1ms — deliberately too short to succeed
try:
    sock.connect(('10.255.255.1', 80))   # a non-routable address
    print("connected (unexpected)")
except socket.timeout:
    print("socket.timeout: timed out")
finally:
    sock.close()

4: settimeout(0.001) caps EVERY blocking socket call — connect, recv — at 1 millisecond.
6: connect() to a non-routable address never gets a response — without a timeout this line blocks indefinitely.
8: The timeout fires as socket.timeout, a normal Python exception the caller can catch and handle.

Output

socket.timeout: timed out

Why this works: The connect() call has no way to know the destination will never answer — 10.255.255.1 is a non-routable address chosen specifically so nothing responds. settimeout(0.001) is what turns that unknown, unbounded wait into a concrete, catchable exception after 1 millisecond, instead of the caller hanging with no way to know whether the call is still in progress or already dead.

Using a client with no timeout at all

Wrong

python

import httpx

with httpx.Client() as client:      # no timeout argument
    response = client.get(slow_url)  # can hang indefinitely

Better

python

import httpx

with httpx.Client(timeout=5.0) as client:   # explicit ceiling
    response = client.get(slow_url)          # fails fast past 5s

What you see: A request thread stays blocked for minutes or hours against a hung downstream, tying up a worker/connection that could be serving other requests — often the actual cause of a cascading outage, not the original slow service alone.

Why: httpx.Client() with no timeout argument waits forever by design — there is no built-in ceiling. One slow or hung downstream then holds your resources hostage indefinitely, which is how a single struggling service takes an entire caller down with it.

A timeout turns silence into a decision point

Call starts

deadline set, e.g. 5s

No response by deadline

downstream may be alive or dead — unknown

Timeout fires

raises an exception NOW, not eventually

Timeout knobs on Python's common HTTP clients

Timeout knobs on Python's common HTTP clients
Client	Setting	Behavior when it fires
socket.socket	sock.settimeout(seconds)	raises socket.timeout on connect or recv
httpx.Client	httpx.Client(timeout=5.0)	raises httpx.TimeoutException
httpx (split)	httpx.Timeout(connect=2.0, read=5.0)	each phase gets its own budget
requests	requests.get(url, timeout=5)	raises requests.exceptions.Timeout

Together

python

import httpx

# connect must succeed within 2s; the response body must arrive within 5s
timeout = httpx.Timeout(connect=2.0, read=5.0)
with httpx.Client(timeout=timeout) as client:
    try:
        response = client.get('https://api.example.com/orders')
    except httpx.TimeoutException:
        response = None  # caller decides what happens next, not the network

Remember: Every network call needs an explicit timeout — no timeout means "wait forever," and a hung downstream then hangs you too.

Retries, exponential backoff, and jitter

coreintermediate

A retry runs a failed call again instead of giving up immediately. Exponential backoff waits longer between each attempt (1s, 2s, 4s...) so a struggling service gets room to recover instead of being hit harder. Jitter adds a random amount to that wait so many clients retrying at once do not all hit the service at the exact same moment.

Think of it as

A crowd that all got turned away at once and agreed to "try again in exactly 10 seconds" arrives back at the door in one simultaneous wave — the same problem that got them turned away, on a timer. Jitter is telling each person a slightly different number near 10 seconds, so they trickle back in instead of slamming the door again all together.

python

import random, time

def retry_with_backoff_and_jitter(fn, max_attempts=5, base=1.0, cap=30.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except TRANSIENT_ERRORS as exc:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))

What we're doing: Retry a call that fails twice then succeeds, with exponential backoff and jitter between attempts, and confirm it recovers instead of failing outright.

retry_backoff_jitter.pypython

import random, time

def call_flaky_service(state={"n": 0}):
    state["n"] += 1
    if state["n"] < 3:
        raise ConnectionError("service unavailable")
    return "ok"

def retry_with_backoff_and_jitter(fn, max_attempts=5, base=0.01, cap=1.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except ConnectionError as exc:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            sleep_for = random.uniform(0, ceiling)
            print(f"attempt {attempt} failed ({exc}); sleeping up to {sleep_for:.4f}s")
            time.sleep(sleep_for)

random.seed(42)
result = retry_with_backoff_and_jitter(call_flaky_service)
print("result:", result)

4: The fake service fails on its first two calls, then succeeds on the third — a realistic transient blip.
13: ceiling doubles each attempt (base * 2^(attempt-1)), capped at cap — this is the exponential backoff.
14: random.uniform(0, ceiling) — jitter — picks a random wait UP TO that ceiling, instead of always waiting exactly the ceiling.

Output

attempt 1 failed (service unavailable); sleeping up to 0.0064s
attempt 2 failed (service unavailable); sleeping up to 0.0005s
result: ok

Why this works: The call fails on attempts 1 and 2 exactly as the fake service is written to, and the loop catches each ConnectionError and sleeps for a random, growing-ceiling delay rather than giving up — by attempt 3 the service succeeds and the loop returns its result instead of raising, which is the entire value a retry loop adds over a single unprotected call.

Retrying with a fixed delay and no jitter

Wrong

python

import time

def retry_fixed_delay(fn, max_attempts=5, delay=2.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except ConnectionError:
            if attempt == max_attempts:
                raise
            time.sleep(delay)   # every client waits exactly 2s, every time

Better

python

import random, time

def retry_with_backoff_and_jitter(fn, max_attempts=5, base=1.0, cap=30.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except ConnectionError:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))

What you see: When a service recovers from an outage, every client that was retrying arrives back at exactly the same moment (they all waited the same fixed delay) — a synchronized "thundering herd" that can knock the just-recovered service back down immediately.

Why: A fixed delay is identical for every client hitting the same failure at roughly the same time, so their retries stay synchronized indefinitely — jitter breaks that synchronization by spreading retries across a random window, and exponential growth gives a sustained outage room to actually recover instead of being retried at full frequency the whole time.

Each failed attempt waits longer, by a randomized amount

Call fails

Attempts left?

Wait (backoff × jitter)

Raise to caller

What each layer of a retry strategy adds

What each layer of a retry strategy adds
Layer	Adds	Without it
Retry	tries the call again after a transient failure	one blip permanently fails the whole operation
Exponential backoff	waits longer each attempt (1s, 2s, 4s, 8s...)	every retry hits the struggling service at full speed immediately
Cap	ceilings the wait (e.g. never more than 30s)	the delay grows unbounded on a long outage
Jitter	randomizes the wait within that ceiling	every client retries in lockstep, recreating the original spike
Max attempts	gives up after N tries and surfaces an error	a dead service is retried forever, hiding the failure from the caller

Together

python

import random, time

def retry_with_backoff_and_jitter(fn, max_attempts=5, base=0.01, cap=1.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except ConnectionError:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))  # jitter within the backoff ceiling

Remember: Retry only transient failures, wait longer each time (exponential backoff), and randomize that wait (jitter) — or every client retries in the same synchronized wave.

Stopping a failure from cascading

A circuit breaker that stops calling a known-failing downstream entirely, self-imposed rate limiting and pool-exhaustion handling, and idempotency as the precondition that makes any of the retrying above actually safe.

Circuit breakers

coreadvanced

A circuit breaker tracks a downstream call's recent failures. After too many in a row, it "opens" and rejects further calls immediately — without even trying — for a cooldown period, then lets one test call through to check if the downstream has recovered.

Think of it as

It works exactly like an electrical circuit breaker: too much current (too many failures) trips it open, cutting the circuit before more damage happens. Nothing flows while it is open. After a cooldown, it lets a small test current through (half-open) — if that succeeds, it closes again; if not, it trips back open.

python

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=30.0):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.failure_count = 0
        self.state = "closed"
        self.opened_at = None

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.reset_timeout:
                self.state = "half-open"
            else:
                raise CircuitOpenError("circuit is open")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failure_count += 1
            if self.state == "half-open" or self.failure_count >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        else:
            self.failure_count, self.state = 0, "closed"
            return result

What we're doing: Drive a real circuit breaker through closed → open (3 failures) → rejected-while-open → half-open trial → closed again, and confirm every transition.

circuit_breaker.pypython

import time

class CircuitOpenError(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=3, reset_timeout=0.05):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.failure_count = 0
        self.state = "closed"
        self.opened_at = None

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.reset_timeout:
                self.state = "half-open"
            else:
                raise CircuitOpenError("circuit is open; call rejected")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failure_count += 1
            if self.state == "half-open" or self.failure_count >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        else:
            self.failure_count, self.state = 0, "closed"
            return result

def unreliable_service(succeed=False):
    if not succeed:
        raise ConnectionError("downstream 500")
    return "ok"

cb = CircuitBreaker(failure_threshold=3, reset_timeout=0.05)
outcomes = []
for _ in range(3):
    try:
        cb.call(unreliable_service, succeed=False)
    except ConnectionError:
        outcomes.append("fail")

try:
    cb.call(unreliable_service, succeed=False)
except CircuitOpenError:
    outcomes.append("rejected-fast")

time.sleep(0.06)
try:
    cb.call(unreliable_service, succeed=True)
    outcomes.append("half-open-success")
except Exception:
    outcomes.append("half-open-fail")

print("outcomes:", outcomes)
print("final state:", cb.state)

9: failure_threshold=3 means the breaker opens after the 3rd consecutive failure.
19: Each failure increments failure_count; once it reaches the threshold, state flips to "open" and the cooldown clock (opened_at) starts.
41: The 4th call arrives while still open and reset_timeout has NOT elapsed yet — it is rejected without ever calling unreliable_service.
46: After sleeping past reset_timeout, the breaker allows exactly one trial call through in half-open state — this one succeeds, so it closes.

Output

outcomes: ['fail', 'fail', 'fail', 'rejected-fast', 'half-open-success']
final state: closed

Why this works: The first three calls genuinely fail and are counted, tripping the breaker open exactly at the threshold. The fourth call is rejected as "rejected-fast" WITHOUT unreliable_service ever running — that is the entire point of open state: no wasted call. After reset_timeout elapses, the breaker allows one real trial call through; it succeeds, so failure_count resets and state returns to closed, matching the state machine exactly.

Circuit breaker state machine

Closed (calls flow)

start

Open (rejects immediately)

Half-open (one trial call)

Confusing a circuit breaker with a retry loop

Wrong

python

# "circuit breaker" that just retries forever with no open state
def call_with_breaker(fn):
    while True:
        try:
            return fn()
        except ConnectionError:
            continue   # keeps calling the dead service indefinitely

Better

python

cb = CircuitBreaker(failure_threshold=3, reset_timeout=30.0)

def call_with_breaker(fn):
    return cb.call(fn)   # stops calling the service once it's known to be down

What you see: Every single request keeps paying the full timeout cost against a service that has been down for the last hour, instead of failing in microseconds once the breaker has already learned it is down.

Why: A retry loop keeps trying the SAME failing call within one request. A circuit breaker remembers failures ACROSS requests and stops attempting the downstream entirely once it is clearly unhealthy — the two solve different problems and a system usually needs both: retries for a single blip, a breaker for a sustained outage.

Circuit breaker states

Circuit breaker states
State	Calls allowed?	Transitions to
closed	yes, all of them	open, once failure_count reaches the threshold
open	no — rejected immediately, no downstream call made	half-open, once reset_timeout elapses
half-open	yes, exactly one trial call	closed on success; open again on failure

Together

python

cb = CircuitBreaker(failure_threshold=3, reset_timeout=30)

for _ in range(3):
    try:
        cb.call(unreliable_service)
    except ConnectionError:
        pass          # each failure counted; breaker opens on the 3rd

try:
    cb.call(unreliable_service)   # rejected without even calling the service
except CircuitOpenError:
    print("failing fast — no wasted network call")

Remember: Closed calls flow; enough failures open it and calls are rejected immediately with no downstream call; after a cooldown, one half-open trial call decides whether to close again or reopen.

Rate limiting as a client-side reliability defense

standardintermediate

A client can rate-limit ITSELF — capping how fast it sends requests to a downstream — so that when that downstream is struggling, your traffic does not push it further into failure. This is the same token-bucket idea a server uses to protect itself, applied on the calling side instead.

Think of it as

A server-side rate limit is a bouncer at the door, stopping too many people from entering. A client-side rate limit is you personally deciding not to run at that door in the first place, even though nobody has stopped you yet — because you can see the room is already full.

python

import time

class ClientSideLimiter:
    def __init__(self, capacity, refill_rate):
        self.capacity, self.tokens = capacity, capacity
        self.refill_rate = refill_rate
        self.last = time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill_rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

What we're doing: Cap outgoing calls to a downstream with a client-side token bucket so a burst of 5 calls only sends 2 immediately and self-throttles the rest, protecting the downstream instead of the caller.

client_side_limiter.pypython

import time

class ClientSideLimiter:
    def __init__(self, capacity, refill_rate):
        self.capacity, self.tokens = capacity, capacity
        self.refill_rate = refill_rate
        self.last = time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill_rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

limiter = ClientSideLimiter(capacity=2, refill_rate=1)
sent, skipped = 0, 0
for _ in range(5):
    if limiter.allow():
        sent += 1
    else:
        skipped += 1

print(f"sent to downstream: {sent}, self-throttled: {skipped}")

17: capacity=2 means only 2 calls are allowed to burst through immediately; the rest are held back by the client itself.
22: 5 rapid calls with no real time passing between them means only the starting 2 tokens are available — 3 are self-throttled.

Output

sent to downstream: 2, self-throttled: 3

Why this works: The limiter caps outgoing calls at its own capacity regardless of whether the downstream has rejected anything yet — 2 of the 5 rapid calls go out, and 3 are held back by the CLIENT. This is the reliability payoff: the caller reduces its own load on a struggling service proactively, rather than waiting to be told to slow down by a 429 that may arrive only after damage is already done.

Remember: A client can throttle its own calls to a struggling downstream before being told to — the same token-bucket shape as a server rate limit, applied on the calling side to avoid piling on.

Idempotency as what makes a retry safe

standardintermediate

An operation is idempotent if running it twice has the same effect as running it once. Retrying a NON-idempotent operation (like "charge $10") repeats its side effect on every retry — retrying an idempotent one (or one made idempotent with a dedup key) is always safe.

Think of it as

Pressing an elevator call button twice does not summon two elevators — the second press changes nothing, because the button is idempotent. Retrying "withdraw $10 from this account" without protection is instead like handing someone a $10 bill every time you repeat the request — each repeat genuinely happens again.

python

processed = {}

def charge_card(amount, dedup_key):
    if dedup_key in processed:
        return processed[dedup_key]     # already done — return the same result, don't repeat it
    charge_payment_provider(amount)
    processed[dedup_key] = "charged"
    return processed[dedup_key]

What we're doing: Retry the same non-idempotent charge 3 times two ways — without a dedup key and with one — and show the account is only actually debited once in the safe version.

idempotent_retry.pypython

account = {"balance": 100}

def charge_card_unsafe(amount):
    account["balance"] -= amount
    raise TimeoutError("response lost, but the charge went through")

def naive_retry(fn, *args, attempts=3):
    for _ in range(attempts):
        try:
            return fn(*args)
        except TimeoutError:
            continue

naive_retry(charge_card_unsafe, 10, attempts=3)
print("unsafe balance after 3 retries:", account["balance"])

processed = {}
account["balance"] = 100

def charge_card_idempotent(amount, dedup_key):
    if dedup_key in processed:
        return processed[dedup_key]
    account["balance"] -= amount
    processed[dedup_key] = "charged"
    return processed[dedup_key]

for _ in range(3):
    try:
        charge_card_idempotent(10, "req-abc-123")
    except TimeoutError:
        continue
print("idempotent balance after 3 retries:", account["balance"])

4: The unsafe version has no memory of past calls — every retry genuinely debits the account again.
20: The idempotent version checks dedup_key FIRST — a retry with the same key returns the stored result instead of re-running the debit.

Output

unsafe balance after 3 retries: 70
idempotent balance after 3 retries: 90

Why this works: The unsafe function has no way to tell "this is a retry of the same request" from "this is a new request" — every one of the 3 retries genuinely subtracts 10, taking the balance from 100 to 70. The idempotent version checks dedup_key before acting: the first call debits once and records the key; the two retries that follow find the key already processed and return the stored result without touching the balance again, leaving it correctly at 90.

Remember: Retrying a write is only safe if the write is idempotent — a retry loop and a non-idempotent operation without a dedup key is a recipe for duplicated side effects.

Connection pooling as part of the resilience picture

standardintermediate

A connection pool is a fixed number of reusable connections, not an unlimited supply. What happens when every connection is already in use — block and wait, time out, or reject immediately — is a reliability decision every pooled resource needs, independent of the performance win pooling itself provides.

Think of it as

A pool is a small fleet of shared taxis, not one per rider. Reuse is why it is efficient — but a fixed fleet size also means a decision has to be made when every taxi is out: wait for one to come back, give up after a while, or refuse the ride outright. That decision is what keeps a traffic spike from turning into gridlock.

python

class ConnectionPool:
    def __init__(self, size):
        self.size, self.in_use = size, 0

    def acquire(self):
        if self.in_use >= self.size:
            raise TimeoutError("pool exhausted; waited and gave up")
        self.in_use += 1
        return f"conn-{self.in_use}"

    def release(self):
        self.in_use = max(0, self.in_use - 1)

What we're doing: Exhaust a pool of size 2 with a 3rd acquire, confirm it fails as a handled error rather than hanging, then confirm a released connection becomes available again.

pool_exhaustion.pypython

class ConnectionPool:
    def __init__(self, size):
        self.size, self.in_use = size, 0

    def acquire(self):
        if self.in_use >= self.size:
            raise TimeoutError("pool exhausted; waited and gave up")
        self.in_use += 1
        return f"conn-{self.in_use}"

    def release(self):
        self.in_use = max(0, self.in_use - 1)

pool = ConnectionPool(size=2)
c1 = pool.acquire()
c2 = pool.acquire()
try:
    c3 = pool.acquire()
except TimeoutError as e:
    print("3rd acquire on a pool of 2:", e)

pool.release()
c3 = pool.acquire()
print("after releasing one, acquire succeeds:", c3)

6: The pool has a hard ceiling (self.size) — this is what makes it a pool rather than an unlimited connection factory.
16: The 3rd acquire on a size-2 pool fails as a normal, catchable TimeoutError instead of silently blocking forever.

Output

3rd acquire on a pool of 2: pool exhausted; waited and gave up
after releasing one, acquire succeeds: conn-2

Why this works: The pool enforces its fixed size — two connections are acquired successfully, and the third fails immediately as a handled exception because the pool is genuinely full, not because anything is broken. Releasing one connection frees capacity, and the next acquire succeeds — demonstrating that pool exhaustion is a normal, expected, and recoverable condition a caller needs to plan for, not an edge case to ignore.

Remember: A pool is finite by design — decide upfront what happens when it is exhausted (fail fast with a timeout, not block forever) rather than discovering it under load.

Boundaries and graceful degradation

Rejecting a malformed or oversized request before it reaches business logic, isolating resource pools per dependency, and degrading to a worse-but-working fallback instead of failing outright.

Request validation and input limits

standardintermediate

Request validation rejects a malformed or out-of-range request before it reaches business logic. Input limits are validation applied to SIZE — a maximum string length, list length, or payload size — so a request cannot exhaust memory or processing time just by being enormous.

Think of it as

A bouncer checking IDs at the door is validation — wrong shape, turned away immediately. A venue's fire-code capacity limit is an input limit — even a perfectly valid guest gets turned away once the room is full, because "correct" and "safe to admit unlimited amounts of" are different questions.

python

MAX_ITEMS = 100
MAX_NAME_LEN = 200

def validate_order(payload):
    errors = []
    name = payload.get("customer_name", "")
    if not name or len(name) > MAX_NAME_LEN:
        errors.append(f"customer_name must be 1-{MAX_NAME_LEN} characters")
    items = payload.get("items", [])
    if not items or len(items) > MAX_ITEMS:
        errors.append(f"items must be 1-{MAX_ITEMS} entries")
    if errors:
        raise ValueError("; ".join(errors))
    return {"customer_name": name, "items": items}

What we're doing: Validate an order payload against both a field-length limit and a list-length limit, rejecting each with a specific message, and accept one within bounds.

validate_order.pypython

MAX_ITEMS = 3
MAX_NAME_LEN = 20

def validate_order(payload):
    errors = []
    name = payload.get("customer_name", "")
    if not isinstance(name, str) or not name:
        errors.append("customer_name is required")
    elif len(name) > MAX_NAME_LEN:
        errors.append(f"customer_name exceeds {MAX_NAME_LEN} characters")
    items = payload.get("items", [])
    if not isinstance(items, list) or not items:
        errors.append("items must be a non-empty list")
    elif len(items) > MAX_ITEMS:
        errors.append(f"items exceeds limit of {MAX_ITEMS}")
    if errors:
        raise ValueError("; ".join(errors))
    return {"customer_name": name, "items": items}

try:
    validate_order({"customer_name": "A" * 30, "items": ["sku1"]})
except ValueError as e:
    print("rejected:", e)

try:
    validate_order({"customer_name": "river", "items": ["sku1", "sku2", "sku3", "sku4"]})
except ValueError as e:
    print("rejected:", e)

print("accepted:", validate_order({"customer_name": "river", "items": ["sku1", "sku2"]}))

1: MAX_ITEMS and MAX_NAME_LEN are the input limits — explicit ceilings, not left unbounded.
9: The length check runs only after confirming name is a non-empty string — order matters, or len() on the wrong type raises the wrong error.
21: A 30-character name exceeds the 20-character limit and is rejected with a specific, field-named message.

Output

rejected: customer_name exceeds 20 characters
rejected: items exceeds limit of 3
accepted: {'customer_name': 'river', 'items': ['sku1', 'sku2']}

Why this works: Each bad payload is rejected for the specific reason it is bad — a name over the length limit, then a list over the item limit — with an error naming the exact field and rule violated. The final payload satisfies both checks and is returned unchanged, showing that validation only blocks what actually violates a rule rather than second-guessing valid input.

Validating shape but leaving size unbounded

Wrong

python

def parse_items(raw_items):
    return [str(x) for x in raw_items]   # correct types, but no cap on how many

Better

python

def parse_items(raw_items, max_items=1000):
    out = []
    for i, x in enumerate(raw_items, start=1):
        if i > max_items:
            raise ValueError(f"too many items: limit is {max_items}")
        out.append(str(x))
    return out

What you see: A request with 200,000 items passes type validation (every item IS a valid string) and still gets fully processed — consuming memory and CPU proportional to whatever size the caller chose to send, with no ceiling.

Why: Checking that each element has the right TYPE says nothing about how many elements are allowed. A caller (malicious or just buggy) can send an arbitrarily large list and it will pass type-only validation every time — the size limit is a separate, mandatory check, not something type validation does for free.

Remember: Validate shape at the boundary AND cap size (string length, list length, payload size) — a request can be perfectly well-typed and still unbounded.

Bulkheads

standardadvanced

A bulkhead gives each downstream dependency (or feature) its own separate, limited pool of resources — threads, connections — instead of sharing one pool across everything. If one dependency saturates its own pool, the others keep working because they were never sharing it.

Think of it as

A ship's hull is divided into sealed compartments so one puncture floods only that compartment, not the whole ship. A bulkhead in software is the same idea applied to resource pools: the "reports" feature getting a flood of slow requests should not sink "checkout" too, if they never shared a pool to begin with.

python

class Bulkhead:
    def __init__(self, name, capacity):
        self.name, self.capacity, self.in_use = name, capacity, 0

    def run(self, fn):
        if self.in_use >= self.capacity:
            raise RuntimeError(f"{self.name} bulkhead full; rejecting immediately")
        self.in_use += 1
        try:
            return fn()
        finally:
            self.in_use -= 1

reports_pool = Bulkhead("reports", capacity=5)
checkout_pool = Bulkhead("checkout", capacity=20)   # separate, unaffected by reports

What we're doing: Saturate the "reports" bulkhead and confirm calls through the separate "checkout" bulkhead still succeed, because the two never share a pool.

bulkheads.pypython

class Bulkhead:
    def __init__(self, name, capacity):
        self.name, self.capacity, self.in_use = name, capacity, 0

    def run(self, fn):
        if self.in_use >= self.capacity:
            raise RuntimeError(f"{self.name} bulkhead full; rejecting immediately")
        self.in_use += 1
        try:
            return fn()
        finally:
            self.in_use -= 1

reports_pool = Bulkhead("reports", capacity=1)
checkout_pool = Bulkhead("checkout", capacity=2)

# Simulate the reports bulkhead already saturated by a slow in-flight call
reports_pool.in_use = 1
try:
    reports_pool.run(lambda: "second report")
except RuntimeError as e:
    print("reports bulkhead:", e)

result = checkout_pool.run(lambda: "checkout-ok")
print("checkout still works while reports is saturated:", result)

14: reports_pool and checkout_pool are entirely separate Bulkhead instances — no shared state between them.
17: Manually setting in_use=1 simulates a slow report already occupying the only reports slot (capacity=1).
21: checkout_pool.run succeeds because it has its own capacity=2 and has never been touched by the reports saturation.

Output

reports bulkhead: reports bulkhead full; rejecting immediately
checkout still works while reports is saturated: checkout-ok

Why this works: The reports bulkhead is genuinely full and rejects the new call immediately, exactly like an exhausted pool should. Checkout succeeds despite that, because checkout_pool tracks its OWN in_use count — the two pools share no state, so saturating one has zero effect on the other. That isolation is the entire value a bulkhead adds over one shared pool for everything.

Remember: Give each dependency its own resource pool — a bulkhead — so one saturated dependency cannot starve every other feature sharing your service.

Graceful degradation and fallbacks

coreintermediate

Graceful degradation means a failing dependency makes your response WORSE, not broken — a fallback is the specific worse-but-working substitute you return instead: cached data instead of live, a generic list instead of personalized, a default instead of a computed value.

Think of it as

A dimmer switch degrades gracefully — losing power reduces the light, it does not plunge the room into total darkness the instant a single bulb fails. A fallback is choosing what the dimmer switch REVERTS to: a safe, known-good default the room can live with while the failure gets fixed.

python

def get_recommendations():
    try:
        return get_recommendations_from_ml_service()
    except (ConnectionError, TimeoutError):
        return get_fallback_recommendations()   # degraded, but the page still works

What we're doing: Show a recommendation service call falling back to a static bestseller list on failure, so the page still renders something useful instead of an error.

graceful_degradation.pypython

def get_recommendations_from_ml_service():
    raise ConnectionError("recommendation service down")

def get_fallback_recommendations():
    return ["bestseller-1", "bestseller-2", "bestseller-3"]

def get_recommendations():
    try:
        return get_recommendations_from_ml_service()
    except ConnectionError:
        return get_fallback_recommendations()

print("recommendations shown to user:", get_recommendations())

1: The primary dependency is down — this simulates a real ML/recommendation service outage.
9: Only ConnectionError is caught — the exact failure mode the fallback is designed for, not every possible exception.
10: The fallback returns a static, always-available list — worse than a real recommendation, but never empty or broken.

Output

recommendations shown to user: ['bestseller-1', 'bestseller-2', 'bestseller-3']

Why this works: get_recommendations_from_ml_service() genuinely raises, simulating the dependency being down — but the caller never sees an exception or a broken page, because the except clause catches exactly that failure and substitutes a static, always-available list. The user gets a working — if generic — page instead of an error, which is the entire purpose of a fallback.

A fallback that catches every exception, hiding real bugs as if they were outages

Wrong

python

def get_recommendations():
    try:
        return get_recommendations_from_ml_service()
    except Exception:                    # catches EVERYTHING, including bugs
        return get_fallback_recommendations()

Better

python

def get_recommendations():
    try:
        return get_recommendations_from_ml_service()
    except (ConnectionError, TimeoutError):   # only the actual failure modes
        return get_fallback_recommendations()

What you see: A genuine bug — a TypeError from bad data, a ZeroDivisionError from a miscalculated denominator — silently returns the fallback list instead of raising, so the bug ships and stays invisible in logs and monitoring, indistinguishable from a real outage.

Why: except Exception matches any error, not just the downstream-failure ones a fallback is meant to absorb. A bug inside get_recommendations_from_ml_service unrelated to network/availability gets silently treated as "the service is down," hiding it behind a fallback that was never meant to mask actual defects — only genuine unavailability.

A failed dependency degrades the response, not the whole request

Call recommendation service

Fails (timeout/down)

Return fallback list

Response still succeeds

What degrades to what

What degrades to what
Ideal	Degraded fallback	Why this is an acceptable trade
Live personalized recommendations	Generic bestseller list	still useful; not personalized
Fresh database read	Last-known cached value	possibly stale, but not empty
Real-time inventory count	Last-synced count, or "in stock"	occasional oversell risk, acceptable at low volume
Full search with ranking service	Plain substring match	less relevant results, still functional

Remember: A fallback catches only the specific failure it is meant to handle and substitutes a worse-but-working result — a bare except in a fallback hides real bugs as if they were outages.