beginnerintermediateadvanced

Django · Section 59

Redis for Django

Level: advanced
Read: 30 min
Concepts: 3

Redis values have types, and the type decides both which commands exist and what they cost. Strings hold bytes and give you atomic `INCR`; hashes let one field be written without reading the object; lists are O(1) at both ends, which is what makes them queues; sets give O(1) membership and set algebra; and sorted sets attach a float score to each member, which — scored by time — turns into sliding-window rate limits, delayed queues and leaderboards with no scan. Expiry is per key, never per hash field, so per-item TTLs mean one key each or a time-scored sorted set trimmed by score. "Atomic" in Redis means exactly one command: `INCR`, `HSET` and `SET … NX EX` are safe under concurrency while `GET` then `SET` is not, and a Lua script is how you make several steps indivisible, because the server runs the whole script as one command. For messaging the two options differ in guarantee rather than style — Pub/Sub delivers only to whoever is connected right now, with no retention, acknowledgement or replay, while Streams retain entries until acknowledged, track each consumer group's position, and let a restarted consumer reclaim its pending work. Finally, one Redis usually plays four roles at once — cache, rate limiter, Celery broker, distributed lock — and each needs its failure behaviour chosen deliberately: the cache should fail open to the database, rate limiting is fail-open or fail-closed per endpoint, a broker outage should be an honest 503 plus a retryable row, and a leased lock can expire under a live-but-slow holder, so it reduces duplicate work rather than guaranteeing exclusion.

Django overview

Python Django Full Course for Beginners | Complete All-in-One Tutorial | 3 HoursDave Gray

What is true here

The value type decides the commands and the cost; sorted sets solve more Django problems than people expect.
Expiry is per key — a hash field cannot have its own TTL.
"Atomic" means one command; use a Lua script when the indivisible unit is larger.
Pub/Sub has no retention or acknowledgement; Streams have both, plus replay and consumer groups.
A TTL-based lock can expire while its holder is alive, so correctness must live in the database.

What you will be able to do

Choose the Redis type that makes the operation you actually perform cheap
Write concurrency-safe Redis code by reasoning about what is one command and what is two
Pick between Pub/Sub and Streams from the delivery guarantee the work requires
Decide, in advance and per role, what your application does when Redis is unavailable

One Redis instance, four roles — and what each one owes you when it fails

Django process

Cache · db 1

strings and hashes, TTL per key

Rate limiter · db 2

a sorted set scored by timestamp

Celery broker · db 0

lists, consumed by workers

Distributed lock · db 2

SET … NX EX, released by token

Fail open → the database

slower pages, not an outage

Fail open or closed, per endpoint

search: open. payments: closed.

503 + a row a sweeper retries

never a silently dropped job

Unique constraint / conditional UPDATE

where the real guarantee lives

The data types

Strings, hashes, lists, sets and sorted sets — and the Django-shaped problem each one makes cheap.

Strings, hashes, lists, sets, and sorted sets

coreintermediate

Redis is not a key-value store where every value is a blob — the value has a *type*, and the type decides which commands are available and what they cost. A **string** holds bytes and supports atomic `INCR`. A **hash** holds field-value pairs, so you can read or write one field without fetching the whole object. A **list** is an ordered sequence with cheap pushes and pops at either end, which is what makes it a queue. A **set** holds unique members with O(1) membership tests and set algebra. A **sorted set** gives every member a numeric score and keeps them ordered by it, which is how you build a leaderboard or a sliding-window rate limiter. Picking the right type is most of what "using Redis well" means.

Think of it as

The reason the type matters is that Redis is single-threaded for command execution, so every command you run is time nobody else's command gets. That turns type choice into a latency decision rather than a style preference. Storing an object as a JSON string means every field update is read-modify-write across the network, and two concurrent updaters lose one of their changes; storing it as a hash makes `HSET user:42 last_seen …` a single atomic field write with no read at all. The same reasoning explains why the collection types are worth learning rather than reaching for a string every time: `SISMEMBER` on a set is one round trip and constant time, while "fetch a JSON array and check in Python" is the whole array over the wire on every check. Sorted sets are the type most people under-use and the one that solves the most Django-shaped problems — a leaderboard is `ZREVRANGE`, a delayed queue is `ZRANGEBYSCORE` on a timestamp score, and a sliding-window rate limiter is `ZADD` plus `ZREMRANGEBYSCORE` plus `ZCARD`, all without a single scan.

python

r = redis.Redis.from_url(settings.REDIS_URL)
r.hset("user:42", mapping={"plan": "pro", "seats": 5})
r.zadd("leaderboard", {"ana": 1450})

What we're doing: A sliding-window rate limiter in a sorted set — exact, with no scan and no background cleanup job.

common/ratelimit.pypython

def allow(user_id, limit=100, window=60):
    key = f"rl:{user_id}"
    now = time.time()

    pipe = r.pipeline()
    pipe.zremrangebyscore(key, 0, now - window)   # drop events older than the window
    pipe.zadd(key, {str(uuid4()): now})           # record this attempt
    pipe.zcard(key)                               # how many remain in the window
    pipe.expire(key, window)                      # the key cleans itself up
    _, _, count, _ = pipe.execute()

    return count <= limit


def window_reset(user_id, window=60):
    oldest = r.zrange(f"rl:{user_id}", 0, 0, withscores=True)
    return int(oldest[0][1] + window - time.time()) if oldest else 0

6: Trimming by score *is* the expiry. Because the score is a timestamp, "older than the window" is a range delete rather than a scan over members.
7: A unique member per attempt. Using the timestamp itself as the member would collapse two requests in the same instant into one recorded event.
9: `EXPIRE` on the whole key means an idle user's data disappears on its own — no cleanup job, no growing keyspace.
5–10: A pipeline sends all four commands in one round trip. Redis executes commands one at a time, so this is also close to atomic for a single key.
16: The oldest score plus the window is exactly when a slot frees up — which is what a `Retry-After` header should say.

Why this works: A fixed-window counter in a string resets on a boundary and lets a caller send twice the limit across it. A sorted set keyed on time is a true sliding window, and trimming by score keeps it bounded without any periodic job.

Storing an object as a JSON string and updating one field

Wrong

python

profile = json.loads(r.get(f"user:{uid}"))
profile["last_seen"] = time.time()
r.set(f"user:{uid}", json.dumps(profile))
# Two concurrent requests: both read, both write, one update is lost.

Better

python

r.hset(f"user:{uid}", "last_seen", time.time())   # one atomic field write

What you see: `last_seen` and `seats` occasionally revert to older values under concurrent traffic. Nothing errors, and it is unreproducible in a single-threaded test.

Why: A JSON string forces read-modify-write for every change, and the read and the write are separate commands with a gap between them. A second writer reading in that gap bases its write on stale data and overwrites the first change on `SET`. A hash makes the field the unit of update, so `HSET` writes only what changed and no read is involved at all.

Five types, and what each one makes cheap

String and Hash

INCR is atomic

a counter with no read-modify-write race

HSET writes one field

no read, no lost update from a concurrent writer

JSON-in-a-string loses both

every field change becomes read-modify-write

List and Set

LPUSH / BRPOP

O(1) at both ends, and BRPOP blocks — a work queue

SISMEMBER is O(1)

membership without moving the collection to Python

LRANGE 0 -1 is O(n)

a list is a queue, not something to scan repeatedly

Sorted set

ZREVRANGE 0 9

a top-ten leaderboard, already sorted

score = a timestamp

ZRANGEBYSCORE gives a sliding window or a delayed queue

ZADD + ZREMRANGEBYSCORE + ZCARD

an exact sliding-window rate limiter, no scan

The five types, and the Django-shaped problem each one solves

The five types, and the Django-shaped problem each one solves
Type	Key commands	Solves
String	`SET`, `GET`, `INCR`, `SETEX`	a cached value; an atomic counter
Hash	`HSET`, `HGET`, `HGETALL`, `HINCRBY`	an object whose fields update independently
List	`LPUSH`, `RPOP`, `BRPOP`, `LLEN`	a work queue; a bounded activity log
Set	`SADD`, `SISMEMBER`, `SINTER`, `SCARD`	membership; tags; "who is online"
Sorted set	`ZADD`, `ZREVRANGE`, `ZRANGEBYSCORE`	leaderboards; sliding windows; delayed jobs

Together

bash

ZADD leaderboard 1450 "ana" 1320 "raj"
ZREVRANGE leaderboard 0 9 WITHSCORES     # top ten, already sorted

Remember: The value has a type, and the type decides both the commands and the cost. `INCR` on a string and `HSET` on a hash field are atomic single writes — JSON in a string turns every change into a lossy read-modify-write. Lists are O(1) at the ends and O(n) to scan, so they are queues. Sets give O(1) membership. Sorted sets are the one to reach for more often: score by time and you get sliding windows, delayed work, and leaderboards without a scan. And never `KEYS` in production — Redis is single-threaded, so it blocks everyone.

Expiry, atomicity, and messaging

TTLs on keys, what "one command" guarantees, and the delivery gap between Pub/Sub and Streams.

TTL and expiration, atomic commands, Pub/Sub, and Streams

coreadvanced

Expiry in Redis is set per key, not per field: `EXPIRE key seconds` or `SET key value EX 300`, and `TTL key` tells you what is left. A key with no TTL lives until something deletes it or memory pressure evicts it. Atomicity comes from Redis executing one command at a time — so any *single* command is atomic, which is why `INCR`, `SETNX` and `ZADD` are safe under concurrency while a read followed by a write is not; `MULTI`/`EXEC` and Lua scripts extend that to a group. For messaging there are two very different tools: **Pub/Sub** is fire-and-forget broadcast — a subscriber that is offline misses the message forever — while **Streams** are an append-only log with consumer groups, acknowledgements and replay, which is what you need when losing a message matters.

Think of it as

Two things here look like features and are really guarantees you have to reason about. The first is that expiry is a property of the *key*: you cannot expire one field of a hash, so any design that wanted per-item TTLs inside a collection needs either one key per item or a sorted set scored by time, where trimming by score does the job an expiry would. The second is that "atomic" in Redis means "one command", and that single sentence resolves most concurrency questions here — `GET` then `SET` is two commands with a gap, `INCR` is one with no gap, and when your logic genuinely needs several steps to be indivisible the answer is a Lua script, because Redis runs the whole script as one command. Pub/Sub versus Streams is the other decision people get wrong, and the honest framing is delivery guarantee: Pub/Sub delivers to whoever is connected *right now* and to nobody else, with no acknowledgement, no retention and no replay — which is exactly right for a cache-invalidation ping or a live dashboard tick, and exactly wrong for anything you would be upset to lose. Streams keep the message until it is acknowledged, track which consumer has seen what, and let you recover a crashed consumer's pending entries, which is the behaviour people wrongly expect from Pub/Sub.

python

if r.set(f"lock:{key}", token, nx=True, ex=30):   # set-if-absent + TTL, one command
    try:
        ...
    finally:
        r.delete(f"lock:{key}")

What we're doing: A distributed lock that a crashed holder cannot keep, released only by its owner.

common/locks.pypython

RELEASE = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""


@contextmanager
def redis_lock(name, ttl=30):
    token = uuid4().hex
    acquired = r.set(f"lock:{name}", token, nx=True, ex=ttl)
    if not acquired:
        raise LockUnavailable(name)
    try:
        yield
    finally:
        # Only delete if we still hold it — a plain DELETE could remove
        # a lock a LATER holder acquired after our TTL expired.
        r.eval(RELEASE, 1, f"lock:{name}", token)

1–6: A Lua script runs as a single command, so the compare and the delete cannot be interleaved. This is the general answer whenever "atomic" needs to span more than one operation.
12: A unique token per acquisition. It is what makes ownership checkable — without it, a release cannot tell "my lock" from "someone else's lock with the same name".
13: `nx=True, ex=ttl` in one `SET`. Acquiring and then setting the expiry separately leaves a window where a crash between them strands the lock forever.
21: The failure this prevents: your work overran the TTL, Redis expired the lock, another worker took it, and a plain `DELETE` would now release *their* lock.

Why this works: The TTL stops a crashed holder from blocking everyone; the token plus the Lua compare-and-delete stops the TTL from turning into a different bug where one worker releases another's lock.

Acquiring a lock with `SETNX`, then setting the TTL separately

Wrong

python

if r.setnx(f"lock:{name}", 1):
    r.expire(f"lock:{name}", 30)    # a crash between these two lines
    ...                             # leaves a lock with NO expiry, held forever

Better

python

if r.set(f"lock:{name}", token, nx=True, ex=30):
    ...

What you see: A job stops running entirely, and it stays stopped across restarts and deploys. `TTL lock:name` returns `-1` — the key exists with no expiry, and only a manual `DEL` brings the job back.

Why: Two commands mean a gap, and a process killed in that gap has created a lock nothing will ever release. `SET … NX EX` makes acquisition and expiry a single command, so the key either exists with a TTL or does not exist at all. This is the same "atomic means one command" rule that makes `INCR` safe and `GET`-then-`SET` unsafe.

The same event, published and streamed — with one consumer restarting

Producer

Pub/Sub channel

Stream

Consumer

1. SUBSCRIBE order-events
2. PUBLISH order-events {"id": 57}
3. delivered — no ack expected
4. consumer restarts (deploy)the subscription is gone with the connection
5. PUBLISH order-events {"id": 58}
6. zero subscribers — the message is discardedno error, no retention, no replay
7. XADD orders * id 58
8. restarts, XREADGROUP … >reads what it has not yet acknowledged
9. 58 delivered, and 57 still pending if it was never XACKed
10. XACK after the work commits

Pub/Sub or Streams?

Pub/Sub or Streams?
Property	Pub/Sub	Streams
Delivery if the consumer is down	lost forever	retained until acknowledged
Acknowledgement	none	`XACK`, with pending-entry tracking
Replay / history	none	yes — read from any id
Multiple consumers	every subscriber gets a copy	consumer groups split the work
Fits	cache-invalidation pings, live tickers	anything you would be upset to lose

Together

bash

XADD orders '*' event created id 57
XREADGROUP GROUP workers w1 COUNT 10 STREAMS orders '>'
XACK orders workers 1725441000-0

Remember: TTL belongs to the key, never to a hash field — for per-item expiry use one key each or a time-scored sorted set. "Atomic" in Redis means "one command": `INCR` and `SET … NX EX` are safe, `GET` then `SET` is not, and a Lua script is how you make several steps into one. Pub/Sub is fire-and-forget with no retention, acknowledgement or replay, so a subscriber that is down misses the message permanently; Streams retain, acknowledge and let a crashed consumer reclaim its pending entries.

Redis in the stack

Cache, rate limiter, broker and lock — and what each one should do when Redis is not there.

Cache, rate limiter, broker, lock — and what breaks when Redis is down

coreadvanced

One Redis instance usually ends up doing four jobs in a Django stack: the **cache** backend, the **rate-limit** counter store, the **Celery broker**, and a **distributed lock**. They look similar and have completely different consequences when Redis goes away. Losing the cache should degrade you to "slow but correct". Losing the rate limiter means either every request is refused or none are, depending on how you wrote the failure path. Losing the broker means tasks cannot be enqueued at all. And a distributed lock is the one that is not merely unavailable but *unsafe*: a lock with a TTL can expire while you are still working, so two workers can hold it at once — which is why Redis locks are advisory, not a substitute for a database constraint.

Think of it as

Decide the failure behaviour for each role *before* Redis is down, because the default is almost never what you want. The useful question is "fail open or fail closed?" and the answer differs per role: a cache should fail open — serve from the database and carry on — while a rate limiter guarding a payment endpoint arguably should fail closed, and one guarding a search box should not. Nothing decides this for you; an unhandled `ConnectionError` from the cache backend will surface as a 500 on a page that could have rendered perfectly well. The lock caveat deserves separate attention because it is a correctness issue rather than an availability one. A TTL exists so a crashed holder cannot block forever, which necessarily means the lock can expire while its holder is alive but slow — a long GC pause, a stalled network call — and at that moment a second worker legitimately acquires it. No amount of tuning removes this; it is inherent to lease-based locking. So use a Redis lock to *reduce duplicate work*, and put the actual correctness guarantee where it belongs: a unique constraint, `select_for_update()`, or an idempotent operation that does not care how many times it runs.

python

CACHES = {"default": {"BACKEND": "...redis.RedisCache",
                      "LOCATION": REDIS_URL + "/1",
                      "OPTIONS": {"IGNORE_EXCEPTIONS": True}}}   # django-redis: fail open

What we're doing: Make each Redis role fail the way that role should, rather than the way an unhandled exception decides.

common/resilience.pypython

def cached_or_compute(key, compute, timeout=300):
    try:
        value = cache.get(key)
    except RedisError:
        return compute()                 # cache down -> slow but correct
    if value is None:
        value = compute()
        with suppress(RedisError):
            cache.set(key, value, timeout)
    return value


def allow_request(user_id, scope):
    try:
        return _sliding_window_allows(user_id, scope)
    except RedisError:
        # Fail CLOSED on money, OPEN on everything else.
        logger.error("ratelimit_unavailable", extra={"scope": scope})
        return scope not in {"payments", "password_reset"}


def enqueue_report(report_id):
    try:
        build_report.delay(report_id)
    except OperationalError:             # kombu: the broker is unreachable
        Report.objects.filter(pk=report_id).update(status="queued_pending")
        raise ServiceUnavailable("Reports are temporarily unavailable.")


def process_once(order_id):
    with suppress(LockUnavailable), redis_lock(f"order:{order_id}"):
        charge_order(order_id)           # idempotency key inside — the lock only
                                         # reduces duplicate work, it does not guarantee it

2–5: The cache failure path returns the computed value rather than re-raising. A Redis outage becomes a latency problem instead of an outage of your own.
16–19: The rate limiter's fallback is a policy decision written down in code. Leaving it to an unhandled exception picks "fail closed for everything", which takes the whole site down with Redis.
25–27: A broker outage marks the row so a periodic sweeper can re-enqueue, and tells the caller honestly. Swallowing this would silently drop the report.
30–33: The lock is best-effort. `charge_order` carries its own idempotency key, so the two-holders case ends in a refused duplicate rather than a double charge.

Why this works: Each role gets the failure mode it deserves, decided in advance. The alternative is that a single `ConnectionError` propagates from whichever call site happens to run first and turns a degraded dependency into a total outage.

Treating a Redis lock as a correctness guarantee

Wrong

python

with redis_lock(f"order:{order_id}", ttl=30):
    if not Order.objects.get(pk=order_id).is_charged:
        charge_customer(order_id)        # a 35-second stall here = two charges
        Order.objects.filter(pk=order_id).update(is_charged=True)

Better

python

with transaction.atomic():
    updated = (Order.objects
               .filter(pk=order_id, is_charged=False)
               .update(is_charged=True))     # conditional update: exactly one winner
if updated:
    charge_customer(order_id)                # with an idempotency key

What you see: A customer is charged twice, roughly once a month, always during a period of high load or a slow upstream call — and the lock code is demonstrably correct when you read it.

Why: The TTL that stops a crashed holder from blocking forever also means the lock can expire while its holder is alive and merely slow. At that instant a second worker acquires it legitimately, and both are inside the critical section with no error anywhere. Lease-based locks cannot avoid this. Moving the decision into a conditional `UPDATE` — which the database evaluates atomically and which exactly one caller can win — puts the guarantee somewhere that has no timeout.

A leased lock, and the moment two workers hold it

Free — no holder

start

Held by worker A · TTL 30s

A is alive but stalled (GC, slow API call)

TTL expired — Redis considers it free

A and B both believe they hold it

The database refuses the duplicate

end

Released by its owner (compare-and-delete)

end

Four roles, four different failure behaviours

Four roles, four different failure behaviours
Role	Redis is down	What you should do	Isolate with
Cache	every `get` raises	fail open — read from the database	db 1, `KEY_PREFIX`
Rate limiting	counters unavailable	decide per endpoint: open for search, closed for payments	db 2
Celery broker	`delay()` raises	return 503, or write a row a sweeper retries	db 0
Distributed lock	cannot acquire	fall back to doing the work idempotently	db 2

Together

python

try:
    value = cache.get(key)
except RedisError:
    logger.warning("cache_unavailable", exc_info=True)
    value = None          # fall through to the database — slow, but correct

Remember: One Redis usually serves four roles, and each needs its failure mode chosen in advance: the cache should fail open to the database, rate limiting is fail-open or fail-closed *per endpoint*, and a broker outage should be an honest 503 plus a row a sweeper can retry. Keep them in separate databases, because `cache.clear()` flushes the whole database it points at. And a leased lock can expire under a live-but-slow holder, so two workers really can hold it — use it to reduce duplicate work, and put correctness in a unique constraint, a conditional update, or an idempotency key.