Caching

Key features

• A cache trades memory (and staleness risk) for avoiding repeated slow work — a database query, an API call, or a CPU-bound computation
• An in-memory cache lives inside the running process — a plain dict, or functools.lru_cache for function results
• In-memory cache is the fastest option (no network hop) but is per-process — a second process or server has its own, empty cache
• A process restart or deploy wipes an in-memory cache entirely — nothing survives it
• lru_cache(maxsize=N) evicts the least-recently-used entry once full, bounding memory automatically

Command

from functools import lru_cache

@lru_cache(maxsize=128)
def slow_square(x):
    return x * x

Key features

• Redis runs as a separate server process — reached over the network, not accessed as an in-process object
• Because it is separate from any one app process, every app instance shares the same cached data — unlike a per-process in-memory cache
• Values are commonly strings, but Redis also has native hash, list, set, and sorted-set types for structured data
• SET key value EX 60 sets a key with a 60-second TTL directly, using Redis's own built-in expiry
• A network hop to Redis is still far faster than a database query, but strictly slower than an in-process dict lookup

Command

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
r.set("user:42:name", "Alice", ex=300)   # ex=300 seconds TTL
name = r.get("user:42:name")
r.delete("user:42:name")

Key features

• Cache-aside: application checks cache first; on a miss, reads the database and populates the cache — the most common pattern
• Write-through: every write goes to the database AND the cache, in the same request — reads are always fresh, writes are slower
• Write-back (write-behind): writes go to the cache immediately and are flushed to the database later, asynchronously or batched
• Write-back is fastest for writes but risks data loss if the cache fails before the delayed flush completes
• Cache-aside never populates the cache on write — a stale cache entry can persist until it expires or is explicitly invalidated

Command

# Cache-aside read
def get(key):
    if key in cache: return cache[key]
    value = db[key]; cache[key] = value
    return value

# Write-through write
def write_through(key, value):
    db[key] = value; cache[key] = value

# Write-back write
def write_back(key, value):
    cache[key] = value; dirty_queue.append(key)

Key features

• TTL (time-to-live) is a duration after which a cache entry is treated as expired, without any write needing to trigger it
• A read against an expired TTL entry is normally treated as a miss — the cache repopulates it from the source of truth
• Invalidation removes or overwrites a specific cache entry immediately, typically right after the matching database write
• Relying on TTL alone means stale data can be served for up to the full TTL window after a real change
• Relying on invalidation alone means an entry with no matching write path (or a missed one) can live forever

Command

class TTLCache:
    def __init__(self):
        self._store = {}   # key -> (value, expires_at)

    def set(self, key, value, ttl_seconds):
        self._store[key] = (value, time.monotonic() + ttl_seconds)

    def get(self, key):
        value, expires_at = self._store.get(key, (None, 0))
        if time.monotonic() > expires_at:
            self._store.pop(key, None)
            return None
        return value

Key features

• A stampede happens specifically when many concurrent requests hit the SAME expired or missing key at the same moment
• Without protection, every one of those requests independently redoes the full expensive rebuild (a DB query, an API call, a computation)
• A per-key lock lets the first request rebuild while the others block, then all read the now-cached value once it is ready
• The lock must be re-checked AFTER acquiring it — otherwise a second thread that was waiting also redoes the rebuild once the lock is free
• A short random jitter added to each TTL spreads out expirations, so many keys are less likely to expire in the same instant

Command

import threading

locks = {}
locks_guard = threading.Lock()

def get_with_lock(key):
    if key in cache:
        return cache[key]
    with locks_guard:
        lock = locks.setdefault(key, threading.Lock())
    with lock:
        if key in cache:      # re-check inside the lock
            return cache[key]
        value = expensive_rebuild()
        cache[key] = value
        return value

Key features

• Warming runs a script or startup hook that populates known-hot keys before real traffic hits the cache
• It matters most right after a deploy or restart, when an in-memory cache is completely empty (see cold start)
• Warming targets the highest-value keys — top products, popular pages — not the entire dataset, which would be wasteful
• Without warming, the first requests after a restart all pay a cache miss simultaneously — which can itself trigger a stampede
• A read-through cache can approximate warming automatically, but an explicit warm-up script gives control over exactly what loads first

Command

def warm_cache_on_startup():
    for key, value in load_top_products().items():
        cache[key] = value

# called once, at process startup, before serving requests
warm_cache_on_startup()

Key features

• Distributed caching spreads keys across multiple nodes, so total cache capacity is not limited to one machine's memory
• Consistent hashing maps each key to a specific node, so adding or removing a node only reshuffles a fraction of keys, not all of them
• Redis Cluster is the common way to run Redis itself as a distributed cache, sharding keys across multiple Redis nodes
• Replication (a primary plus replicas per shard) adds availability — a replica can serve reads or be promoted if the primary fails
• A distributed cache trades single-node simplicity for horizontal capacity and fault tolerance, at the cost of operational complexity

Command

from redis.cluster import RedisCluster

rc = RedisCluster(host="localhost", port=6379)
rc.set("user:42:name", "Alice", ex=300)
name = rc.get("user:42:name")
# the client library routes each key to the correct node automatically

Key features

• Strong consistency means a read after a write always returns the new value — expensive, since every write must reach the cache before any read can proceed
• Eventual consistency means a read may return a stale value for a bounded window, then eventually reflects the write — this is what TTL-based caching gives you
• Write-through gets close to strong consistency because cache and database update together, in the same request
• Cache-aside with a long TTL and no invalidation accepts the largest inconsistency window, in exchange for the simplest write path
• The right consistency level is a product decision, not a technical default — a stock counter tolerates staleness a payment balance cannot

Command

# stronger consistency: write-through, cache always matches db
def write_through(key, value):
    db[key] = value
    cache[key] = value

# weaker consistency: cache-aside + TTL, bounded staleness window
def get_cache_aside(key):
    if key in cache: return cache[key]
    value = db[key]; cache[key] = value
    return value

Key features

• A cache is meant to be an optimization, not a dependency the application cannot function without
• Failing open catches a cache error and falls back to the source of truth, at the cost of slower requests during the outage
• Failing closed returns an error to the caller instead of touching the database — appropriate only if the database cannot absorb the full traffic the cache normally shields it from
• A cache outage with fail-open still risks a stampede — every request suddenly hits the database directly, same as a mass simultaneous miss
• Timeouts on the cache client matter as much as error handling — a slow-but-not-failed cache can block requests longer than skipping it outright

Command

def get_with_fallback(key, source_of_truth):
    try:
        return cache_get(key)
    except CacheUnavailable:
        return source_of_truth[key]   # fail open