beginnerintermediateadvanced

Python · Section 27

Caching

Level: advanced
Read: 150 min
Concepts: 9

Why caching is needed, in-memory and Redis caches, the three canonical write patterns (cache-aside, write-through, write-back), TTL and invalidation for keeping entries fresh, cache stampede protection, cache warming, distributed caching, consistency trade-offs, and how a cache should fail.

This section

Speeding Up Python Code With CachingNeuralNine

What is true here

A cache trades memory and staleness risk for skipping repeated slow work — an in-memory cache is fastest but per-process; Redis is shared but network-accessed.
Cache-aside fills the cache on a read miss; write-through fills it on every write; write-back fills it on write but delays the database write until later.
TTL expires an entry automatically after a fixed duration; invalidation removes it immediately on a matching write — most real caches need both.
A cache stampede is many concurrent requests rebuilding the same expired key at once — a per-key lock with a re-check inside collapses that to one rebuild.
A cache outage should degrade latency, not correctness — fail open, falling back to the source of truth, and catch only the cache-specific error.

What you will be able to do

Explain why a cache is used and choose between an in-process cache and a shared cache like Redis
Implement cache-aside, write-through, and write-back, and state the trade-off each one makes
Give a cache entry a TTL and invalidate it explicitly on a matching write
Prevent a cache stampede with a per-key lock that re-checks the cache after acquiring it
Warm a cache proactively at startup, and design fail-open behavior for a cache outage
Reason about consistency and distribution trade-offs when a single cache node is not enough

Cache fundamentals

Why a cache exists at all, and the two places one commonly lives — inside the process, or in a shared server like Redis.

Why caching is needed, and in-memory cache

standardbeginner

A cache stores the result of a slow operation so the next request for the same thing skips the work. An in-memory cache keeps that result inside the running process — a dict, or functools.lru_cache — so lookups cost a hash, not a database query or a computation.

Think of it as

Caching is a sticky note with an answer already written on it, next to the phone. Without it, you redial and ask the same question every time. The note only helps as long as it stays true — that tension (fast, but possibly stale) is the whole subject of this section.

python

from functools import lru_cache

@lru_cache(maxsize=128)
def slow_square(x):
    return x * x

What we're doing: Show that lru_cache actually skips re-running the function body on a repeated call, and reports the hit/miss counts.

in_memory_cache.pypython

from functools import lru_cache

calls = {"n": 0}

@lru_cache(maxsize=128)
def slow_square(x):
    calls["n"] += 1
    return x * x

print(slow_square(4))
print(slow_square(4))   # cached — the function body does not run again
print(slow_square(5))
print(calls["n"])
print(slow_square.cache_info())

5: @lru_cache(maxsize=128) wraps slow_square so results are stored by argument, up to 128 distinct arguments.
6: calls["n"] increments only when the wrapped body actually runs — a cache hit skips this entirely.
11: slow_square(4) again returns the stored result — calls["n"] does not increment a second time.

Output

16
16
25
2
CacheInfo(hits=1, misses=2, maxsize=128, currsize=2)

Why this works: The function body ran only twice (calls["n"] == 2) even though slow_square was called three times — the second call with 4 was served from lru_cache's internal dict without re-executing the function, which is exactly what CacheInfo(hits=1, misses=2, ...) confirms.

Caching a plain dict with no size limit

Wrong

python

cache = {}

def get_user(user_id):
    if user_id not in cache:
        cache[user_id] = fetch_user_from_db(user_id)
    return cache[user_id]

# every distinct user_id ever requested stays in memory forever

Better

python

from functools import lru_cache

@lru_cache(maxsize=10_000)
def get_user(user_id):
    return fetch_user_from_db(user_id)

# oldest-used entries are evicted once 10,000 distinct users are cached

What you see: Memory usage grows without bound as more distinct keys are seen — eventually an OOM kill in a long-running process, with no error until then.

Why: A plain dict used as a cache has no eviction policy — every new key adds an entry that is never removed. lru_cache(maxsize=N) bounds memory by discarding the least-recently-used entry once the cache is full.

Remember: A cache saves a slow result in a fast place; an in-memory cache is the fastest kind but lives only in the current process and vanishes on restart.

Redis cache

standardintermediate

Redis is an in-memory key-value store that runs as its own server, separate from your application process. Multiple app processes or servers share the same Redis cache, unlike an in-memory dict that only one process can see.

Think of it as

An in-memory dict is a sticky note on your own desk — only you can read it. Redis is a shared whiteboard in the hallway — every process, on every server, reads and writes the same board, at the cost of a network hop to reach it.

python

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
r.set("user:42:name", "Alice", ex=300)   # ex=300 seconds TTL
name = r.get("user:42:name")
r.delete("user:42:name")

Remember: Redis is a shared, network-accessed cache every app instance can read and write — an in-memory dict cannot be shared across processes or servers.

Patterns and freshness

The three canonical patterns for when a cache is populated relative to a write, and the two mechanisms — TTL and invalidation — for getting a stale entry out.

Cache-aside, write-through, and write-back

coreintermediate

These three patterns differ in when the cache gets filled relative to the database. Cache-aside fills it lazily on a read miss. Write-through fills it immediately on every write, alongside the database. Write-back fills it on write but delays the database write until later.

Think of it as

Think of the cache as a filing cabinet next to a slower archive room. Cache-aside: you only file a copy when someone asks for it and it's not already filed. Write-through: every time you update the archive, you update the filing cabinet in the same trip. Write-back: you update the filing cabinet now and promise to update the archive later, in a batch.

python

# Cache-aside read
def get(key):
    if key in cache: return cache[key]
    value = db[key]; cache[key] = value
    return value

# Write-through write
def write_through(key, value):
    db[key] = value; cache[key] = value

# Write-back write
def write_back(key, value):
    cache[key] = value; dirty_queue.append(key)

What we're doing: Implement all three patterns against plain dicts standing in for the cache and database, and show write-back genuinely delays the database write until flush_dirty() runs.

caching_patterns.pypython

db = {"user:1": "Alice", "user:2": "Bob"}
cache = {}

def get_user_cache_aside(key):
    if key in cache:
        return cache[key]
    value = db[key]
    cache[key] = value
    return value


db2 = {}
cache_wt = {}

def write_through(key, value):
    db2[key] = value
    cache_wt[key] = value


db3 = {}
cache_wb = {}
dirty_queue = []

def write_back(key, value):
    cache_wb[key] = value
    dirty_queue.append(key)

def flush_dirty():
    while dirty_queue:
        k = dirty_queue.pop(0)
        db3[k] = cache_wb[k]


print(get_user_cache_aside("user:1"))
write_through("user:3", "Carol")
print(db2["user:3"], cache_wt["user:3"])
write_back("user:4", "Dave")
print("db3 before flush:", db3)
flush_dirty()
print("db3 after flush:", db3)

4: get_user_cache_aside only reads db — it never writes to it, and only populates cache after a miss.
14: write_through writes db2 and cache_wt in the same function call — both are always in sync.
25: write_back writes cache_wb immediately but only queues the key — db3 is not touched yet.
29: flush_dirty is what actually writes db3, run separately (and later) from write_back itself.

Output

Alice
Carol Carol
db3 before flush: {}
db3 after flush: {'user:4': 'Dave'}

Why this works: db3 before flush: {} proves write_back really did delay the database write — cache_wb already had the value, but db3 stayed empty until flush_dirty() ran explicitly, which is the entire trade write-back makes: faster writes, at the cost of a window where the database and cache disagree.

Using write-back for data that cannot tolerate loss

Wrong

python

# payment record written via write-back
def record_payment(payment_id, amount):
    cache[payment_id] = amount
    dirty_queue.append(payment_id)
    return "accepted"   # confirmed to the caller before the DB has it

Better

python

# payment record written via write-through
def record_payment(payment_id, amount):
    db[payment_id] = amount     # durable before responding
    cache[payment_id] = amount
    return "accepted"

What you see: If the process crashes after queuing but before flush_dirty() runs, a payment the caller was told is "accepted" never reaches the database — silent, unrecoverable data loss.

Why: Write-back only guarantees the database write eventually happens if the process survives long enough to flush the queue. For data where "accepted" must mean durably stored — payments, orders — write-through (or a synchronous write) is required instead.

Cache-aside vs. write-through (the two most-compared patterns)

Cache-aside

+Write goes to the database only
+Cache is filled lazily, on the next read miss
+Simple, and the default choice for most reads

Write-through

−Write goes to the database and the cache together
−Every subsequent read is guaranteed fresh
−Every write pays the extra cache-write latency

Cache-aside vs. write-through vs. write-back

Cache-aside vs. write-through vs. write-back
Pattern	On write	On read	Risk
Cache-aside	writes DB only; cache untouched	check cache, else read DB then populate cache	stale cache until read-through or TTL expiry
Write-through	writes DB and cache together	always reads cache — it is always fresh	every write pays the cache-write cost too
Write-back	writes cache now, DB later (async)	reads cache — fresh, since cache led the write	DB write lost if cache fails before flush

Together

python

# cache-aside: populate only on a miss
def get_user_cache_aside(key):
    if key in cache:
        return cache[key]
    value = db[key]
    cache[key] = value
    return value

# write-through: cache and DB updated together
def write_through(key, value):
    db[key] = value
    cache[key] = value

# write-back: cache updated now, DB flushed later
def write_back(key, value):
    cache[key] = value
    dirty_queue.append(key)

Remember: Cache-aside fills the cache on a read miss; write-through fills it on every write; write-back fills it on write but delays the database write until later.

TTL and cache invalidation

coreintermediate

TTL (time-to-live) makes a cache entry expire automatically after a fixed duration. Invalidation removes an entry immediately, on demand, usually because the underlying data just changed. A cache needs one or both — otherwise stale data lives forever.

Think of it as

TTL is a carton of milk with a printed expiry date — it goes bad on its own, on schedule, whether or not anyone checks. Invalidation is throwing the milk out today because you just poured a fresh carton — an explicit action, tied to the exact moment the old one stopped being true.

python

class TTLCache:
    def __init__(self):
        self._store = {}   # key -> (value, expires_at)

    def set(self, key, value, ttl_seconds):
        self._store[key] = (value, time.monotonic() + ttl_seconds)

    def get(self, key):
        value, expires_at = self._store.get(key, (None, 0))
        if time.monotonic() > expires_at:
            self._store.pop(key, None)
            return None
        return value

What we're doing: Build a minimal TTL cache and prove an entry actually stops being returned once its time-to-live has elapsed.

ttl_cache.pypython

import time

class TTLCache:
    def __init__(self):
        self._store = {}  # key -> (value, expires_at)

    def set(self, key, value, ttl_seconds):
        self._store[key] = (value, time.monotonic() + ttl_seconds)

    def get(self, key):
        if key not in self._store:
            return None
        value, expires_at = self._store[key]
        if time.monotonic() > expires_at:
            del self._store[key]
            return None
        return value


ttl_cache = TTLCache()
ttl_cache.set("session:abc", "user-42", ttl_seconds=0.2)
print(ttl_cache.get("session:abc"))
time.sleep(0.3)
print(ttl_cache.get("session:abc"))

7: set() stores expires_at as an absolute time — now plus ttl_seconds — not the duration itself.
12: get() compares the current time against expires_at on every read, not just at insert.
20: Immediately after set(), 0.2s has not elapsed yet, so the entry is still returned.
22: After sleeping 0.3s (past the 0.2s TTL), get() finds the entry expired and returns None — a live miss.

Output

user-42
None

Why this works: The same key returns its value immediately, then None after sleeping past the TTL — proof the expiry check runs on every get() using a real timestamp comparison, not a one-time timer that silently keeps serving stale data.

Storing the TTL duration instead of the absolute expiry time

Wrong

python

def set(self, key, value, ttl_seconds):
    self._store[key] = (value, ttl_seconds)   # stores the duration, not a deadline

def get(self, key):
    value, ttl_seconds = self._store[key]
    # no way to know how much of ttl_seconds has already elapsed

Better

python

def set(self, key, value, ttl_seconds):
    self._store[key] = (value, time.monotonic() + ttl_seconds)  # absolute deadline

def get(self, key):
    value, expires_at = self._store[key]
    if time.monotonic() > expires_at:
        return None

What you see: Entries never expire, or expire based on the wrong reference point — the duration alone has no way to tell how much time has actually passed since it was set.

Why: A duration is meaningless without a start point. Storing time.monotonic() + ttl_seconds captures a fixed deadline at insert time, so every later get() can compare "now" against that one fixed value.

Two ways a stale entry stops being served

TTL expiry

entry auto-expires after a fixed duration

Explicit invalidation

entry removed the moment a write changes it

Next read

miss — repopulated from the source of truth

TTL vs. explicit invalidation

TTL vs. explicit invalidation
Mechanism	Trigger	Staleness window	Failure mode if skipped
TTL	a fixed duration elapses	up to the full TTL, even after data changes	none — always expires eventually
Invalidation	an explicit write/delete call	near zero — cleared the moment data changes	stale forever if the invalidation call is missed

Together

python

# TTL: expires on its own after ttl_seconds
ttl_cache.set("session:abc", "user-42", ttl_seconds=0.2)

# Invalidation: removed immediately on the matching write
def update_price(product_id, new_price):
    db[f"product:{product_id}"] = new_price
    cache.pop(f"product:{product_id}", None)

Remember: TTL expires an entry automatically after a fixed duration; invalidation removes it immediately on a matching write — most real caches need both.

Operating a cache at scale

What happens under concurrent load, at startup, across multiple nodes, and when the cache itself is unavailable — the operational half of caching, beyond just picking a pattern.

Cache stampede

coreadvanced

A cache stampede happens when a popular cache entry expires and many concurrent requests all miss at once, each independently redoing the same expensive rebuild. A per-key lock — letting only the first request rebuild while the rest wait — prevents it.

Think of it as

Picture one popular item out of stock, and ten shoppers, each one separately walking to the stockroom to check, instead of one shopper checking while the other nine wait at the counter. A stampede is ten expensive rebuilds happening at once for one expired key; a lock makes it one rebuild and nine waiters.

python

import threading

locks = {}
locks_guard = threading.Lock()

def get_with_lock(key):
    if key in cache:
        return cache[key]
    with locks_guard:
        lock = locks.setdefault(key, threading.Lock())
    with lock:
        if key in cache:      # re-check inside the lock
            return cache[key]
        value = expensive_rebuild()
        cache[key] = value
        return value

What we're doing: Run 10 concurrent threads against the same missing key, first with no protection, then with a per-key lock, and count how many times the expensive rebuild actually runs.

cache_stampede.pypython

import threading
import time

db_calls = {"n": 0}

def expensive_rebuild():
    db_calls["n"] += 1
    time.sleep(0.05)
    return "computed-value"


cache_stampede = {}

def get_no_lock(key):
    if key in cache_stampede:
        return cache_stampede[key]
    value = expensive_rebuild()
    cache_stampede[key] = value
    return value


locks = {}
locks_guard = threading.Lock()
cache_locked = {}

def get_with_lock(key):
    if key in cache_locked:
        return cache_locked[key]
    with locks_guard:
        lock = locks.setdefault(key, threading.Lock())
    with lock:
        if key in cache_locked:
            return cache_locked[key]
        db_calls["n"] += 1
        time.sleep(0.05)
        value = "computed-value"
        cache_locked[key] = value
        return value


threads = [threading.Thread(target=get_no_lock, args=("hot_key",)) for _ in range(10)]
for t in threads: t.start()
for t in threads: t.join()
print("rebuilds without lock:", db_calls["n"])

14: get_no_lock checks the cache and, on a miss, rebuilds immediately — nothing stops two threads from both seeing a miss at once.
27: get_with_lock acquires a lock specific to this key before rebuilding, forcing other threads for the SAME key to wait.
31: The re-check inside the lock is what makes this correct — a thread that waited finds the value already cached and returns it instead of rebuilding again.

Output

rebuilds without lock: 10
rebuilds with lock: 1

Why this works: Ten threads racing an unprotected miss produced ten rebuilds — every single one paid the expensive_rebuild() cost. The lock-protected version produced exactly one — the other nine threads blocked on the lock, then found the value already cached by the re-check, proving the lock plus re-check together eliminate the stampede rather than just reducing it.

Locking without re-checking the cache inside the lock

Wrong

python

def get_with_lock(key):
    if key in cache:
        return cache[key]
    with lock:
        value = expensive_rebuild()   # no re-check — every waiter rebuilds too
        cache[key] = value
        return value

Better

python

def get_with_lock(key):
    if key in cache:
        return cache[key]
    with lock:
        if key in cache:              # re-check: filled while this thread waited
            return cache[key]
        value = expensive_rebuild()
        cache[key] = value
        return value

What you see: The lock serializes the rebuilds instead of eliminating them — still N rebuilds for N waiting threads, just one at a time instead of concurrently; slower AND no less expensive in total.

Why: Acquiring the lock only guarantees exclusive access — it does not check whether the work is already done. Without re-checking the cache immediately after acquiring the lock, every thread that was waiting proceeds to redo the rebuild once its turn comes.

Ten concurrent misses on one expired key, with a per-key lock

10 requests miss

same key, same instant

Acquire lock(key)

only one thread gets in first

Re-check cache

waiters find it already filled

Rebuild once

one DB query / API call

Unprotected vs. lock-protected cache read under concurrency

Unprotected vs. lock-protected cache read under concurrency
Approach	What happens on a stampede	Rebuild count for 10 concurrent misses
No lock	every thread sees the miss and rebuilds independently	10
Per-key lock, no re-check	first thread rebuilds; waiters proceed to rebuild too once unblocked	up to 10
Per-key lock + re-check inside	first thread rebuilds; waiters re-check, find it cached, reuse it	1

Together

python

def get_with_lock(key):
    if key in cache:
        return cache[key]
    with locks_guard:
        lock = locks.setdefault(key, threading.Lock())
    with lock:
        if key in cache:          # re-check: another thread may have filled it
            return cache[key]
        value = expensive_rebuild()
        cache[key] = value
        return value

Remember: A stampede is many concurrent requests all rebuilding the same expired key at once — a per-key lock with a re-check inside collapses that to one rebuild.

Cache warming

standardintermediate

Cache warming pre-populates a cache with expected data before traffic arrives — usually at deploy or startup — instead of waiting for the first real request to trigger a slow miss.

Think of it as

A cold cache is an empty shop on opening morning — the first customers each wait while stock is fetched from the back. Warming is stocking the shelves before the doors open, so the very first customer gets a fast response instead of paying the miss cost.

python

def warm_cache_on_startup():
    for key, value in load_top_products().items():
        cache[key] = value

# called once, at process startup, before serving requests
warm_cache_on_startup()

What we're doing: Populate a cache from a known "top products" source before any request arrives, and confirm the cache is empty beforehand and full afterward.

cache_warming.pypython

def load_top_products():
    return {"product:1": "Widget", "product:2": "Gadget", "product:3": "Gizmo"}


warm_cache = {}

def warm_cache_on_startup():
    for key, value in load_top_products().items():
        warm_cache[key] = value


print(len(warm_cache))
warm_cache_on_startup()
print(len(warm_cache))
print(sorted(warm_cache.keys()))

1: load_top_products stands in for a real query — the top N most-requested keys, known ahead of time.
7: warm_cache_on_startup runs once, before any real request is served, filling the cache proactively.

Output

0
3
['product:1', 'product:2', 'product:3']

Why this works: The cache holds 0 entries before warm_cache_on_startup() runs and 3 after — proof the population happened proactively, from a known list, rather than being triggered by an actual incoming request.

Remember: Warming fills a cache with known-hot data before traffic arrives, so the first real requests after a restart do not all pay a miss at once.

Distributed caching

standardadvanced

A distributed cache spreads data across multiple cache nodes instead of one server, so the dataset can exceed a single machine's memory and survive one node failing. A client library decides which node holds a given key.

Think of it as

A single Redis instance is one filing cabinet. A distributed cache is many filing cabinets in a row, with a rule (consistent hashing) that tells you exactly which cabinet a given folder lives in — so you never have to search all of them.

python

from redis.cluster import RedisCluster

rc = RedisCluster(host="localhost", port=6379)
rc.set("user:42:name", "Alice", ex=300)
name = rc.get("user:42:name")
# the client library routes each key to the correct node automatically

Remember: A distributed cache shards keys across multiple nodes via consistent hashing, trading single-node simplicity for capacity beyond one machine and tolerance of a node failing.

Cache consistency

standardadvanced

Cache consistency is how closely a cached value matches the current value in the source of truth. No caching strategy gives perfect real-time consistency — every one of them accepts some window where the two can disagree.

Think of it as

Consistency is asking: how far apart are the two clocks in your house right now? Write-through keeps them within a second of each other, always in sync at the cost of extra writes. Cache-aside with a long TTL can let them drift for minutes. Neither clock is ever wrong forever — the question is how long the drift window is, and whether that window is acceptable for what the data is used for.

python

# stronger consistency: write-through, cache always matches db
def write_through(key, value):
    db[key] = value
    cache[key] = value

# weaker consistency: cache-aside + TTL, bounded staleness window
def get_cache_aside(key):
    if key in cache: return cache[key]
    value = db[key]; cache[key] = value
    return value

Remember: Every caching strategy accepts some staleness window — write-through keeps it near zero, cache-aside with a long TTL accepts minutes; pick based on what the data is used for.

Cache failure behavior

standardadvanced

When the cache itself is unreachable, the application must decide: fail open (fall back to the database, slower but working) or fail closed (return an error instead of hitting the database). Most caches should fail open.

Think of it as

A cache going down should feel like a sticky note falling off the desk, not the phone line going dead. Failing open means you just redial (hit the database) instead of refusing to answer the question at all.

python

def get_with_fallback(key, source_of_truth):
    try:
        return cache_get(key)
    except CacheUnavailable:
        return source_of_truth[key]   # fail open

What we're doing: Show a cache read that raises an error still returns a correct result, by falling back to the source of truth instead of propagating the failure.

fail_open.pypython

class CacheUnavailable(Exception):
    pass

def broken_cache_get(key):
    raise CacheUnavailable("connection refused")

def get_with_fallback(key, source_of_truth):
    try:
        return broken_cache_get(key)
    except CacheUnavailable:
        return source_of_truth[key]   # fail open: fall back to DB

db = {"config:limit": 100}
print(get_with_fallback("config:limit", db))

4: broken_cache_get simulates a real cache outage by always raising CacheUnavailable.
9: The except clause catches specifically CacheUnavailable, not every exception — an unrelated bug should still surface.
10: On catching the cache error, the function reads from source_of_truth instead of propagating the failure to the caller.

Output

Why this works: The call succeeds and returns 100 even though broken_cache_get always raises — the caller never sees the cache outage at all, which is exactly what "fail open" means: the cache being down degrades latency, not correctness.

Catching every exception, not just the cache-specific one

Wrong

python

def get_with_fallback(key, source_of_truth):
    try:
        return cache_get(key)
    except Exception:               # hides real bugs, not just outages
        return source_of_truth[key]

Better

python

def get_with_fallback(key, source_of_truth):
    try:
        return cache_get(key)
    except CacheUnavailable:        # only the specific, expected failure
        return source_of_truth[key]

What you see: A genuine bug in cache_get (a TypeError from bad input, for example) silently falls back to the database instead of surfacing — the bug goes unnoticed and the database absorbs load that should have been an alert.

Why: A bare except Exception catches every failure, expected or not, and treats all of them the same way. Catching the specific CacheUnavailable (or the client library's specific connection-error type) lets an unrelated bug propagate and be seen, instead of being silently masked as "the cache is just down".

Remember: A cache outage should degrade performance (fail open, fall back to the database), not correctness — catch only the specific cache error, not every exception.