API Reliability

Key features

• Every network call needs an explicit timeout — the default in most Python HTTP clients is "no timeout" (wait forever)
• A timeout that fires raises an exception (socket.timeout, httpx.TimeoutException) — it does not return a partial or empty result silently
• Connect timeout (time to establish the connection) and read timeout (time to receive a response) are separate settings — set both
• A timeout that is too short turns healthy-but-slow calls into false failures; too long defeats the point of having one at all
• A timeout bounds ONE attempt — it says nothing about whether to try again, which is what retries decide

Command

import httpx

with httpx.Client(timeout=5.0) as client:
    try:
        response = client.get(url)
    except httpx.TimeoutException:
        # decide here: retry, fall back, or surface an error
        ...

Key features

• Retry only errors that are plausibly transient (timeouts, connection resets, 502/503/504) — never a 400 or 422, which will fail identically every time
• Exponential backoff multiplies the wait each attempt (e.g. base * 2^attempt), capped at a maximum so it does not grow unbounded
• Jitter picks a random value UP TO the backoff ceiling (or randomizes within a window around it) — it does not replace backoff, it varies it
• A retry loop needs a maximum attempt count or maximum total time — an uncapped loop against a permanently dead service retries forever
• Retrying a non-idempotent operation (like charging a card) without an idempotency key can repeat its side effect — see the idempotency concept

Command

import random, time

def retry_with_backoff_and_jitter(fn, max_attempts=5, base=1.0, cap=30.0):
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except TRANSIENT_ERRORS as exc:
            if attempt == max_attempts:
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))

Key features

• Closed is the normal state — calls go through, and the breaker counts consecutive (or windowed) failures
• Open means calls are rejected immediately, without even attempting the downstream — this is what makes a breaker faster than a timeout under sustained failure
• After a reset_timeout, an open breaker moves to half-open and allows exactly one trial call through
• A successful half-open call closes the breaker (failure count resets); a failed one reopens it and restarts the cooldown
• A breaker protects the CALLER from wasting time and resources on a downstream that is known to be failing — it does not fix the downstream itself

Command

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=30.0):
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.failure_count = 0
        self.state = "closed"
        self.opened_at = None

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.reset_timeout:
                self.state = "half-open"
            else:
                raise CircuitOpenError("circuit is open")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failure_count += 1
            if self.state == "half-open" or self.failure_count >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        else:
            self.failure_count, self.state = 0, "closed"
            return result

Key features

• A struggling downstream often gets WORSE under retry storms — every failed call retried immediately multiplies load exactly when the service can least handle it
• A client-side limiter caps outgoing call rate regardless of whether the downstream has told you to slow down yet
• The same token-bucket shape works on either side: capacity for bursts, a steady refill rate for sustained throughput
• Client-side limiting is a form of self-imposed backpressure — different from a circuit breaker, which stops calling entirely rather than just slowing down
• A 429 response with Retry-After from the server should update your limiter's pace directly — it is the downstream telling you its actual budget

Command

import time

class ClientSideLimiter:
    def __init__(self, capacity, refill_rate):
        self.capacity, self.tokens = capacity, capacity
        self.refill_rate = refill_rate
        self.last = time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill_rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Key features

• GET, PUT, and DELETE are naturally idempotent by HTTP's own definition — repeating them produces the same end state as doing it once
• POST is NOT naturally idempotent — "create an order" run twice creates two orders unless something explicitly prevents that
• A retry loop (this section's own retries-backoff-and-jitter concept) is only safe to point at a non-idempotent operation if that operation has been made idempotent first
• The standard fix is a dedup key: the caller generates an ID once, and the operation checks "have I already done this exact ID?" before acting again
• This is the same idempotency-key MECHANISM api-development covers — the reliability lens is WHY it matters: it is the precondition, not an optional add-on, for retrying a write safely

Command

processed = {}

def charge_card(amount, dedup_key):
    if dedup_key in processed:
        return processed[dedup_key]     # already done — return the same result, don't repeat it
    charge_payment_provider(amount)
    processed[dedup_key] = "charged"
    return processed[dedup_key]

Key features

• Validate at the boundary, before business logic runs — a request that fails validation should never reach a database call or downstream service
• A missing size limit is an availability risk, not just a data-quality one — an unbounded list or string lets one request consume memory proportional to its size
• Reject with a specific, actionable error message per field — "customer_name exceeds 20 characters" is debuggable; "invalid request" is not
• Limits belong on every unbounded input: string length, list/array length, nesting depth, and total payload size
• A limit that is too low breaks legitimate use; too high (or absent) leaves the system exposed to a single oversized request degrading it for everyone

Command

MAX_ITEMS = 100
MAX_NAME_LEN = 200

def validate_order(payload):
    errors = []
    name = payload.get("customer_name", "")
    if not name or len(name) > MAX_NAME_LEN:
        errors.append(f"customer_name must be 1-{MAX_NAME_LEN} characters")
    items = payload.get("items", [])
    if not items or len(items) > MAX_ITEMS:
        errors.append(f"items must be 1-{MAX_ITEMS} entries")
    if errors:
        raise ValueError("; ".join(errors))
    return {"customer_name": name, "items": items}

Key features

• A pool has a maximum size — once every connection is checked out, the next request to acquire one has to wait, time out, or be rejected
• An unbounded pool (or no pool at all — a new connection per request) defeats the reason pools exist and can exhaust the downstream's own connection limit under load
• A blocking acquire with NO timeout on an exhausted pool just moves the "hung forever" problem from the network call to the pool itself
• Sizing a pool too small under-uses available concurrency; too large can overwhelm the downstream service or database it connects to
• Pool exhaustion under load is a signal worth treating like any other reliability failure — surfaced as an error and handled, not silently blocked on indefinitely

Command

class ConnectionPool:
    def __init__(self, size):
        self.size, self.in_use = size, 0

    def acquire(self):
        if self.in_use >= self.size:
            raise TimeoutError("pool exhausted; waited and gave up")
        self.in_use += 1
        return f"conn-{self.in_use}"

    def release(self):
        self.in_use = max(0, self.in_use - 1)

Key features

• Without bulkheads, ALL downstream calls typically share one thread pool or connection pool — one slow dependency can consume the entire shared pool
• A bulkhead gives each dependency its own dedicated pool sized for its own expected load, isolated from every other dependency's pool
• A saturated bulkhead rejects (or queues) new calls to THAT dependency only — every other dependency, with its own separate pool, is unaffected
• Bulkheads are a sizing/isolation decision made upfront, distinct from a circuit breaker (a runtime decision to stop calling a specific failing dependency)
• The trade-off is real: isolated pools mean less resource sharing overall, so a healthy dependency cannot "borrow" idle capacity from another's pool

Command

class Bulkhead:
    def __init__(self, name, capacity):
        self.name, self.capacity, self.in_use = name, capacity, 0

    def run(self, fn):
        if self.in_use >= self.capacity:
            raise RuntimeError(f"{self.name} bulkhead full; rejecting immediately")
        self.in_use += 1
        try:
            return fn()
        finally:
            self.in_use -= 1

reports_pool = Bulkhead("reports", capacity=5)
checkout_pool = Bulkhead("checkout", capacity=20)   # separate, unaffected by reports

Key features

• Graceful degradation is a design goal: the SYSTEM keeps functioning, in a reduced way, instead of failing completely because one piece failed
• A fallback is the concrete implementation — a specific alternate value or behavior returned in place of the one that could not be produced
• A fallback must catch ONLY the specific failure types it is meant to handle (network/timeout errors from that dependency), never a bare except that also swallows real bugs
• The best fallbacks degrade the LEAST important part of the response first — a stale cache beats a missing recommendation, which beats a broken page
• A fallback path needs to be exercised (tested, or triggered in production occasionally) — an untested fallback is often broken exactly when it is finally needed

Command

def get_recommendations():
    try:
        return get_recommendations_from_ml_service()
    except (ConnectionError, TimeoutError):
        return get_fallback_recommendations()   # degraded, but the page still works