Filter concepts by levelShowing all levels.

Django · Redis

Redis for Django

The five data types, expiry and atomicity, Pub/Sub against Streams, and the four roles one Redis plays

  • The value has a typeand the type decides both the available commands and what they cost
  • "Atomic" means one commandINCR and SET…NX EX are safe; GET then SET is a race
  • Pub/Sub loses what nobody is listening forno retention, no ack, no replay — Streams add all three
  • A leased lock can expire under a live holdertwo workers really can hold it — put correctness elsewhere
  • Choose each role's failure mode in advancecache fails open; a rate limiter is per-endpoint; a broker is an honest 503

1.Strings, hashes, lists, sets, and sorted sets

Redis is not a key-value store where every value is a blob — the value has a *type*, and the type decides which commands are available and what they cost. A **string** holds bytes and supports atomic `INCR`. A **hash** holds field-value pairs, so you can read or write one field without fetching the whole object. A **list** is an ordered sequence with cheap pushes and pops at either end, which is what makes it a queue. A **set** holds unique members with O(1) membership tests and set algebra. A **sorted set** gives every member a numeric score and keeps them ordered by it, which is how you build a leaderboard or a sliding-window rate limiter. Picking the right type is most of what "using Redis well" means.

Key features

  • Strings hold bytes up to 512 MB and support atomic `INCR`/`INCRBY`/`DECR` — the counter primitive, with no read-modify-write race
  • Hashes store field-value pairs under one key: `HSET`/`HGET`/`HGETALL`/`HINCRBY`, so one field updates without transferring the object
  • Lists are doubly-linked: `LPUSH`/`RPOP` are O(1) at the ends, `BRPOP` blocks until an element arrives — that pair is a work queue
  • `LRANGE key 0 -1` reads the whole list and is O(n); a list is a queue, not something to scan repeatedly
  • Sets give O(1) `SADD`/`SISMEMBER` plus `SINTER`/`SUNION`/`SDIFF` — membership and set algebra without moving data to Python
  • Sorted sets attach a float score to each member and keep them ordered: `ZADD`, `ZREVRANGE` for leaderboards, `ZRANGEBYSCORE` for time windows and delayed work

The five types, and the Django-shaped problem each one solves

The five types, and the Django-shaped problem each one solves
TypeKey commandsSolves
String`SET`, `GET`, `INCR`, `SETEX`a cached value; an atomic counter
Hash`HSET`, `HGET`, `HGETALL`, `HINCRBY`an object whose fields update independently
List`LPUSH`, `RPOP`, `BRPOP`, `LLEN`a work queue; a bounded activity log
Set`SADD`, `SISMEMBER`, `SINTER`, `SCARD`membership; tags; "who is online"
Sorted set`ZADD`, `ZREVRANGE`, `ZRANGEBYSCORE`leaderboards; sliding windows; delayed jobs
Five types, and what each one makes cheap

String and Hash

INCR is atomic

a counter with no read-modify-write race

HSET writes one field

no read, no lost update from a concurrent writer

JSON-in-a-string loses both

every field change becomes read-modify-write

List and Set

LPUSH / BRPOP

O(1) at both ends, and BRPOP blocks — a work queue

SISMEMBER is O(1)

membership without moving the collection to Python

LRANGE 0 -1 is O(n)

a list is a queue, not something to scan repeatedly

Sorted set

ZREVRANGE 0 9

a top-ten leaderboard, already sorted

score = a timestamp

ZRANGEBYSCORE gives a sliding window or a delayed queue

ZADD + ZREMRANGEBYSCORE + ZCARD

an exact sliding-window rate limiter, no scan

  • One Redis key
  • String and Hash — a value, or an object
    • INCR is atomic — a counter with no read-modify-write race
    • HSET writes one field — no read, no lost update from a concurrent writer
    • JSON-in-a-string loses both — every field change becomes read-modify-write
  • List and Set — ordered, or unique
    • LPUSH / BRPOP — O(1) at both ends, and BRPOP blocks — a work queue
    • SISMEMBER is O(1) — membership without moving the collection to Python
    • LRANGE 0 -1 is O(n) — a list is a queue, not something to scan repeatedly
  • Sorted set — a score per member, kept in order
    • ZREVRANGE 0 9 — a top-ten leaderboard, already sorted
    • score = a timestamp — ZRANGEBYSCORE gives a sliding window or a delayed queue
    • ZADD + ZREMRANGEBYSCORE + ZCARD — an exact sliding-window rate limiter, no scan

Command

python
r = redis.Redis.from_url(settings.REDIS_URL)
r.hset("user:42", mapping={"plan": "pro", "seats": 5})
r.zadd("leaderboard", {"ana": 1450})

The value has a type, and the type decides both the commands and the cost. `INCR` on a string and `HSET` on a hash field are atomic single writes — JSON in a string turns every change into a lossy read-modify-write. Lists are O(1) at the ends and O(n) to scan, so they are queues. Sets give O(1) membership. Sorted sets are the one to reach for more often: score by time and you get sliding windows, delayed work, and leaderboards without a scan. And never `KEYS` in production — Redis is single-threaded, so it blocks everyone.

2.TTL and expiration, atomic commands, Pub/Sub, and Streams

Expiry in Redis is set per key, not per field: `EXPIRE key seconds` or `SET key value EX 300`, and `TTL key` tells you what is left. A key with no TTL lives until something deletes it or memory pressure evicts it. Atomicity comes from Redis executing one command at a time — so any *single* command is atomic, which is why `INCR`, `SETNX` and `ZADD` are safe under concurrency while a read followed by a write is not; `MULTI`/`EXEC` and Lua scripts extend that to a group. For messaging there are two very different tools: **Pub/Sub** is fire-and-forget broadcast — a subscriber that is offline misses the message forever — while **Streams** are an append-only log with consumer groups, acknowledgements and replay, which is what you need when losing a message matters.

Key features

  • TTL is per key: `EXPIRE key 300`, `SET key val EX 300`, `TTL key` (`-1` means no TTL, `-2` means the key does not exist)
  • You cannot expire a single hash field — use one key per item, or a sorted set scored by time and trimmed with `ZREMRANGEBYSCORE`
  • A write command that does not also read is atomic on its own: `INCR`, `SETNX`, `ZADD`, `HSET` — a `GET` then `SET` pair is not
  • `SET key val NX EX 30` is the correct lock acquire: set-if-absent *and* expiry in one command, so a crashed holder cannot hold it forever
  • Pub/Sub is fire-and-forget: no persistence, no acknowledgement, no replay — a subscriber that is down when a message is published never sees it
  • Streams (`XADD`/`XREADGROUP`/`XACK`) retain messages, track per-consumer position, and expose pending entries so a crashed consumer's work can be reclaimed

Pub/Sub or Streams?

Pub/Sub or Streams?
PropertyPub/SubStreams
Delivery if the consumer is downlost foreverretained until acknowledged
Acknowledgementnone`XACK`, with pending-entry tracking
Replay / historynoneyes — read from any id
Multiple consumersevery subscriber gets a copyconsumer groups split the work
Fitscache-invalidation pings, live tickersanything you would be upset to lose
The same event, published and streamed — with one consumer restarting
Producer
Pub/Sub channel
Stream
Consumer
  1. 1SUBSCRIBE order-events
  2. 2PUBLISH order-events {"id": 57}
  3. 3delivered — no ack expected
  4. 4consumer restarts (deploy)the subscription is gone with the connection
  5. 5PUBLISH order-events {"id": 58}
  6. 6zero subscribers — the message is discardedno error, no retention, no replay
  7. 7XADD orders * id 58
  8. 8restarts, XREADGROUP … >reads what it has not yet acknowledged
  9. 958 delivered, and 57 still pending if it was never XACKed
  10. 10XACK after the work commits
  1. Consumer → Pub/Sub channel: SUBSCRIBE order-events
  2. Producer → Pub/Sub channel: PUBLISH order-events {"id": 57}
  3. Pub/Sub channel → Consumer: delivered — no ack expected
  4. Consumer → Pub/Sub channel: consumer restarts (deploy) (the subscription is gone with the connection)
  5. Producer → Pub/Sub channel: PUBLISH order-events {"id": 58}
  6. Pub/Sub channel → Producer: zero subscribers — the message is discarded (no error, no retention, no replay)
  7. Producer → Stream: XADD orders * id 58
  8. Consumer → Stream: restarts, XREADGROUP … > (reads what it has not yet acknowledged)
  9. Stream → Consumer: 58 delivered, and 57 still pending if it was never XACKed
  10. Consumer → Stream: XACK after the work commits

Command

python
if r.set(f"lock:{key}", token, nx=True, ex=30):   # set-if-absent + TTL, one command
    try:
        ...
    finally:
        r.delete(f"lock:{key}")

TTL belongs to the key, never to a hash field — for per-item expiry use one key each or a time-scored sorted set. "Atomic" in Redis means "one command": `INCR` and `SET … NX EX` are safe, `GET` then `SET` is not, and a Lua script is how you make several steps into one. Pub/Sub is fire-and-forget with no retention, acknowledgement or replay, so a subscriber that is down misses the message permanently; Streams retain, acknowledge and let a crashed consumer reclaim its pending entries.

3.Cache, rate limiter, broker, lock — and what breaks when Redis is down

One Redis instance usually ends up doing four jobs in a Django stack: the **cache** backend, the **rate-limit** counter store, the **Celery broker**, and a **distributed lock**. They look similar and have completely different consequences when Redis goes away. Losing the cache should degrade you to "slow but correct". Losing the rate limiter means either every request is refused or none are, depending on how you wrote the failure path. Losing the broker means tasks cannot be enqueued at all. And a distributed lock is the one that is not merely unavailable but *unsafe*: a lock with a TTL can expire while you are still working, so two workers can hold it at once — which is why Redis locks are advisory, not a substitute for a database constraint.

Key features

  • Give each role its own Redis database or key prefix — `cache.clear()` runs `FLUSHDB` and will delete queued Celery tasks if they share a database
  • The cache should fail open: wrap reads in a `try`/`except` so a Redis outage degrades to database reads rather than 500s
  • Decide fail-open versus fail-closed per rate limiter — a payment endpoint and a search box want opposite answers, and neither is the default
  • A TTL-based lock can expire while its holder is still working, so two workers can hold it simultaneously — this is inherent to lease-based locking, not a tuning problem
  • Treat a Redis lock as an optimisation that reduces duplicate work; put correctness in a unique constraint, `select_for_update()`, or an idempotent operation
  • Losing the broker stops `delay()` from enqueuing at all — decide whether that is a 503 to the caller or a row written for a later sweeper to pick up

Four roles, four different failure behaviours

Four roles, four different failure behaviours
RoleRedis is downWhat you should doIsolate with
Cacheevery `get` raisesfail open — read from the databasedb 1, `KEY_PREFIX`
Rate limitingcounters unavailabledecide per endpoint: open for search, closed for paymentsdb 2
Celery broker`delay()` raisesreturn 503, or write a row a sweeper retriesdb 0
Distributed lockcannot acquirefall back to doing the work idempotentlydb 2
A leased lock, and the moment two workers hold it
SET lockNX EX 30work finishesinside the TTLA stalls30s pass —nothing tells AB acquireslegitimatelyunique constraint / idempotencykey stops the second writeA finishes and deletesonly its OWN token

Free — no holder

start

Held by worker A · TTL 30s

A is alive but stalled (GC, slow API call)

TTL expired — Redis considers it free

A and B both believe they hold it

The database refuses the duplicate

end

Released by its owner (compare-and-delete)

end

  • Free — no holder (start)
    • → Held by worker A · TTL 30s when SET lock NX EX 30
  • Held by worker A · TTL 30s
    • → Released by its owner (compare-and-delete) when work finishes inside the TTL
    • → A is alive but stalled (GC, slow API call) when A stalls
  • A is alive but stalled (GC, slow API call)
    • → TTL expired — Redis considers it free when 30s pass — nothing tells A
  • TTL expired — Redis considers it free
    • → A and B both believe they hold it when B acquires legitimately
  • A and B both believe they hold it
    • → The database refuses the duplicate when unique constraint / idempotency key stops the second write
    • → Free — no holder when A finishes and deletes only its OWN token
  • The database refuses the duplicate (end)
  • Released by its owner (compare-and-delete) (end)

Command

python
CACHES = {"default": {"BACKEND": "...redis.RedisCache",
                      "LOCATION": REDIS_URL + "/1",
                      "OPTIONS": {"IGNORE_EXCEPTIONS": True}}}   # django-redis: fail open

One Redis usually serves four roles, and each needs its failure mode chosen in advance: the cache should fail open to the database, rate limiting is fail-open or fail-closed *per endpoint*, and a broker outage should be an honest 503 plus a row a sweeper can retry. Keep them in separate databases, because `cache.clear()` flushes the whole database it points at. And a leased lock can expire under a live-but-slow holder, so two workers really can hold it — use it to reduce duplicate work, and put correctness in a unique constraint, a conditional update, or an idempotency key.

Remember

  • Sorted sets are the under-used type: score by time and you get sliding-window rate limits, delayed queues, and leaderboards without a scan.
  • TTL belongs to the key, not to a hash field — per-item expiry means one key each, or a time-scored sorted set trimmed by ZREMRANGEBYSCORE.
  • Never KEYS in production: Redis executes one command at a time, so it blocks every other client. Use SCAN, and UNLINK for large values.
  • A Redis lock reduces duplicate work; a unique constraint, a conditional UPDATE, or an idempotency key is what actually prevents it.

Pro tip

  • Store objects as hashes, not JSON strings: HSET writes one field atomically, while JSON forces a read-modify-write that loses concurrent updates.
  • Use a Lua script whenever "atomic" must span more than one command — Redis runs the whole script as a single command.

Next up

  • Async Django
  • Background Jobs
  • Celery

Sources

  • Redis data types and patterns — redis.io/docs/latest/develop/
  • Every claim checked against Redis 7.x docs · checked 2026-09-04

Spotted a mistake on this page? Tell us.