Networking and HTTP Clients

Key features

• requests has no built-in async client — every call blocks until the response arrives
• httpx.Client is synchronous like requests; httpx.AsyncClient is its async twin, same method names
• aiohttp.ClientSession is async-only — there is no synchronous aiohttp client
• requests is the most widely installed of the three and has the largest plugin/adapter ecosystem
• httpx supports HTTP/2 (http2=True); requests and aiohttp are HTTP/1.1 only

Command

import requests
r = requests.get(url)              # sync, blocks

import httpx
r = httpx.get(url)                 # sync, requests-like API
async with httpx.AsyncClient() as c:
    r = await c.get(url)           # async, same method names

import aiohttp
async with aiohttp.ClientSession() as s:
    async with s.get(url) as r:    # async, response is a context manager
        data = await r.json()

Key features

• Creating a new requests.Session()/httpx.Client() per call defeats pooling — construct one and reuse it across every request
• timeout=N in requests/httpx applies to the connect AND read phases separately unless split as timeout=(connect, read)
• A request with no timeout= can hang indefinitely — requests defaults to no timeout at all if you do not pass one
• requests' retry logic lives on the transport adapter (urllib3.util.retry.Retry), mounted via HTTPAdapter, not a client kwarg
• httpx's built-in retries= (on HTTPTransport) only retries connection-level failures, never a 4xx/5xx status — status-code retries need custom code

Command

# requests — pool via Session, bound with timeout, retry via adapter
with requests.Session() as session:
    session.mount('https://', HTTPAdapter(max_retries=Retry(total=3, status_forcelist=[502, 503, 504])))
    r = session.get(url, timeout=(3.05, 10))   # (connect timeout, read timeout)

# httpx — pool via Client + Limits, timeout via Timeout(), retries via transport
limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)
with httpx.Client(limits=limits, timeout=httpx.Timeout(10.0, connect=3.05)) as client:
    r = client.get(url)

# aiohttp — pool via TCPConnector, timeout via ClientTimeout
timeout = aiohttp.ClientTimeout(total=10, connect=3.05)
connector = aiohttp.TCPConnector(limit=100, limit_per_host=10)
async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
    async with session.get(url) as r:
        ...

Key features

• Bearer token auth is a plain header: headers={'Authorization': f'Bearer {token}'} — no special client support required
• requests' auth=(username, password) tuple sends HTTP Basic auth automatically; httpx uses auth=httpx.BasicAuth(username, password)
• A custom auth scheme can subclass requests.auth.AuthBase (or pass a callable to httpx's auth=) to inject headers on every request via one client
• Setting default headers once on a Session/Client (session.headers.update({...})) applies them to every request made through it
• Never put a token in the URL query string — it lands in access logs, browser history, and proxy logs; the Authorization header does not

Command

# Bearer token -- a plain header, works identically on any client
headers = {'Authorization': f'Bearer {token}'}

# Basic auth -- requests
requests.get(url, auth=(username, password))

# Basic auth -- httpx
httpx.get(url, auth=httpx.BasicAuth(username, password))

# apply to every request through one reused client
client = httpx.Client(headers={'Authorization': f'Bearer {token}'})

Key features

• requests: pass stream=True to get(), then iterate response.iter_content(chunk_size=N) — the body is NOT downloaded until you iterate
• httpx: use the httpx.stream(method, url) or client.stream(...) context manager, then response.iter_bytes(N) or .iter_text(N)
• aiohttp: iterate response.content.iter_chunked(N) inside async for — .content is a StreamReader, read incrementally by default
• Forgetting to close a streamed response (or not using it as a context manager) leaves the underlying connection open, blocking pool reuse
• response.json() or .text on a streamed response force-reads the WHOLE body first — they defeat the point of streaming

Command

with requests.get(url, stream=True) as r:
    for chunk in r.iter_content(chunk_size=8192):
        f.write(chunk)

with httpx.stream('GET', url) as r:
    for chunk in r.iter_bytes(8192):
        f.write(chunk)

async with session.get(url) as r:
    async for chunk in r.content.iter_chunked(8192):
        f.write(chunk)

Key features

• asyncio.gather(*(client.get(u) for u in urls)) starts every request before waiting on any of them
• Reuse one httpx.AsyncClient (or aiohttp.ClientSession) across all the concurrent calls — a fresh client per call adds a new connection handshake each time
• Total time for N concurrent requests is close to the slowest single request, not the sum of all N — the real payoff over a sync loop
• A single failing request inside gather() raises and cancels the whole group by default — pass return_exceptions=True to collect failures instead of aborting
• Concurrency has a real ceiling — hundreds of simultaneous connections to one host risks hitting server-side rate limits or connection limits

Command

import asyncio
import httpx

async def fetch_all(urls):
    async with httpx.AsyncClient() as client:      # one client, reused
        return await asyncio.gather(
            *(client.get(u) for u in urls),
            return_exceptions=True,                 # don't abort on one failure
        )

results = asyncio.run(fetch_all(urls))

Key features

• response.status_code (requests/httpx) never raises anything by itself — a 500 response is returned normally, just like a 200
• response.raise_for_status() raises only for 4xx/5xx — it does nothing (returns None) for a successful response
• requests raises requests.exceptions.HTTPError; httpx raises httpx.HTTPStatusError — both carry .response with the original response object
• A network-level failure (DNS, refused connection, timeout) raises a DIFFERENT exception than a bad status code — those never even reach raise_for_status()
• httpx.HTTPStatusError distinguishes response.is_client_error (4xx) from response.is_server_error (5xx) as boolean properties

Command

response = client.get(url)
try:
    response.raise_for_status()   # no-op on 2xx, raises on 4xx/5xx
except requests.exceptions.HTTPError as e:   # httpx.HTTPStatusError for httpx
    handle_failure(e.response.status_code)
else:
    data = response.json()

Key features

• response.json() raises requests.exceptions.JSONDecodeError (httpx: json.JSONDecodeError) if the body is not valid JSON at all — an HTML error page from a proxy is a common real cause
• A 200 response with valid JSON can still be missing an expected key — .json()['missing_key'] raises KeyError, not a validation error
• response.json().get('key') returns None silently on a missing key instead of raising — often the WRONG choice when the field is required
• A Pydantic model (or dataclass + manual checks) turns "does this dict have the right shape" into one explicit validation step instead of scattered .get() calls
• Validate the body even on a successful status — a misconfigured API gateway can return 200 with an HTML login page instead of the expected JSON

Command

response = client.get(url)
response.raise_for_status()
try:
    data = response.json()
except requests.exceptions.JSONDecodeError as e:
    raise RuntimeError(f'response was not valid JSON: {e}') from e

required = data['id']   # explicit KeyError if the API changed shape

Key features

• requests.get(url, proxies={"https": "http://proxy.example.com:8080"}) routes the request through that proxy
• httpx.Client(proxy="http://proxy.example.com:8080") does the same for httpx — one proxy= argument, not a scheme-keyed dict
• verify=False on requests or httpx.Client(verify=False) disables certificate verification — the connection is still encrypted, but no longer confirms who is on the other end
• A real self-signed or expired certificate raises SSLCertVerificationError/ConnectError by default — that failure is the protection working, not a bug to silence
• Prefer trusting a specific internal CA bundle (verify="/path/to/ca-bundle.pem") over verify=False when a private/internal service uses a self-signed cert

Command

import httpx

# Proxy
client = httpx.Client(proxy="http://proxy.example.com:8080")

# NEVER do this against a real host -- disables certificate verification entirely
client = httpx.Client(verify=False)

# Prefer this instead, if the target uses a private/internal CA
client = httpx.Client(verify="/path/to/internal-ca-bundle.pem")