beginnerintermediateadvanced

System Design · Section 38

Distributed Locks

Level: advanced
Read: 16 min
Concepts: 3

Distributed coordination is needed only when the invariant being protected spans multiple processes or systems that no single database transaction already covers — most "we need a distributed lock" problems are actually a unique constraint or an atomic compare-and-set in disguise. When a genuine distributed lock is required, it is almost always implemented as a lease that auto-expires, which trades a permanent-deadlock risk for a different one: a holder that overruns its lease before finishing looks identical, to the lock service, to a client that legitimately released it, letting a second client acquire the same lock while the first still believes it holds it. Fencing tokens — a monotonically increasing number checked at the protected resource itself, not at the lock service — are the standard fix, alongside sizing lease TTLs from measured work duration and accounting for clock drift.

This section

Distributed Locks | System design basicsTech Dummies - Narendra Lakshmana Gowda

What is true here

Distributed coordination is needed only when no single database transaction already spans the invariant being protected.
A lease-based lock auto-expires to avoid permanent deadlock on a crash, but that same expiry lets a slow, still-working holder overlap with a new holder.
Fencing tokens — checked at the protected resource, not the lock service — are what actually prevents a stale holder's write from landing, not a longer TTL.
A unique constraint or an atomic UPDATE ... WHERE expresses most "exactly one" or "only if unchanged" invariants in one database round trip, with none of a lock's lease or clock-drift risk.

What you will be able to do

Recognize when a problem needs genuine distributed coordination versus when a single database's own transaction or constraint already covers it
Explain why a lease-based lock does not by itself guarantee mutual exclusion, and what fencing tokens add
Reason about lease TTL sizing, clock-drift assumptions, and process-crash recovery in a lock design
Choose a unique constraint or atomic UPDATE over a distributed lock when the invariant fits in one database operation

Deciding you need one, and what can go wrong

When distributed coordination is genuinely necessary, and the concrete risks — stale locks, crashes, lease timing, clock drift — that come with a lease-based lock.

When distributed coordination is actually needed

coreintermediate

Distributed coordination — a lock held across multiple processes or machines — is needed only when the resource being protected is itself outside any single database that could enforce the invariant directly. If one Postgres instance already holds the data and can express "only one of these" as a transaction or a constraint, that is not a distributed-locking problem at all; it becomes one when multiple independent workers must serialize access to something a single ACID transaction cannot already cover — a shared external resource, a scheduled job that must run on exactly one of many replicas, or a critical section spanning several services and calls.

Think of it as

A single database transaction is like one person checking out a book from a library's own front desk — the desk itself can refuse to hand out a second copy, no extra coordination needed. A distributed lock is like several branch libraries, with no shared front desk, needing to agree by phone call that only one of them lends out the last copy of a book that physically exists in just one place. You only need the phone call when there genuinely is no single front desk that already knows the whole picture.

What we're doing: Tell apart a case that only looks like it needs a distributed lock from one that genuinely does.

text

Scenario: "Only one worker should send the daily
digest email."

Looks like: needs a distributed lock across N worker
processes racing to be the one that sends it.

Check first: is there a database all N workers already
share? If yes —

  INSERT INTO daily_digest_runs (run_date)
  VALUES ('2026-08-22')
  ON CONFLICT (run_date) DO NOTHING
  RETURNING run_date;

  -- exactly one worker's INSERT returns a row;
  -- the rest get zero rows back and skip sending.

No lock service was needed — the shared database's own
uniqueness guarantee already served as the coordination
point, because the invariant ("has today's digest been
claimed yet") lives entirely inside that one database.

12: ON CONFLICT DO NOTHING is the coordination mechanism — the database resolves the race atomically, with no separate lock service in the picture.
15: Exactly one concurrent INSERT wins; every other worker sees zero affected rows and knows it lost — that is the entire "lock" this scenario needed.

Why this works: This is the single most common misjudgment in practice: reaching for a distributed lock (Redis, ZooKeeper, etcd) for a problem that a shared relational database already solves with a unique constraint, because the *workers* are distributed even though the *data* is not.

Standing up a Redis-based lock for a coordination problem the existing database already covers

Wrong

text

# five workers all share one Postgres database,
# but the team adds Redis purely to serialize
# "who sends the daily digest":
lock = redis.set('digest-lock', worker_id, nx=True, px=30000)
if lock:
    send_digest()
    redis.delete('digest-lock')

Better

text

-- same workers, same shared Postgres — express the
-- invariant where the data already lives:
INSERT INTO daily_digest_runs (run_date)
VALUES (CURRENT_DATE)
ON CONFLICT (run_date) DO NOTHING
RETURNING run_date;
-- a returned row means "you won, send it";
-- no rows means someone else already claimed it.

What you see: A postmortem where the digest was sent twice not because the Redis lock logic was wrong, but because Redis itself had a brief outage or failover unrelated to the actual business data — a dependency the design never needed to take on.

Why: Adding Redis here introduces a second system that must stay available and correctly configured for a guarantee the existing database could give for free — every new coordination service is a new thing that can be down, misconfigured, or drift out of sync with the data it is meant to protect.

Does one database already own this invariant?

Shared invariant

multiple processes involved

One DB already owns it

use a transaction/constraint

Spans systems

needs a distributed lock

Deciding whether a problem needs a distributed lock

Deciding whether a problem needs a distributed lock
Situation	Coordination needed?	Why
Two web servers both writing to the same Postgres row	No	A transaction, unique constraint, or row lock inside Postgres already serializes this
Five worker instances running the same cron job, only one should execute it	Yes	No single database transaction spans "did any worker already claim this run" across independent process starts unless one is designated to own that check — a lock (or a leader) is the coordination mechanism
Multiple app servers calling a rate-limited third-party API that has no server-side idempotency	Yes	The invariant ("no more than N calls per window") lives outside any database the app controls
One microservice validating a request before writing to its own database	No	Single service, single database — ordinary transaction scope covers it

Remember: Ask "does one database already own this invariant?" before reaching for a distributed lock — if yes, use its transaction/constraint machinery; a distributed lock is for coordination that genuinely spans systems no single database transaction covers.

Stale locks, crashes, lease expiration and clock assumptions

coreadvanced

A distributed lock is almost always implemented as a lease — a key that auto-expires after a timeout, so the system does not deadlock forever if the holder crashes. That auto-expiry is also exactly where the risk lives: if the holder is still working when the lease expires (a long garbage-collection pause, a slow network, a scheduler delay), another client can acquire the same lock while the first one believes it still holds it — two processes now both think they have exclusive access at once. Redis's own Redlock documentation is explicit that mutual exclusion is only guaranteed if the holder finishes its work within the lease's validity time minus clock drift, and recommends fencing tokens as the real fix, not just a longer lease.

Think of it as

A lease-based lock is like a parking permit with a printed expiry time instead of a live attendant. If you are still using the spot when the permit expires, the system has no way to know you are still there — it just believes the spot is free again and hands out a new permit to someone else. Both drivers now have a "valid" permit for the same spot at the same time, and neither one did anything wrong; the permit's printed clock was simply not a live measure of who was actually still parked.

What we're doing: Show how a stale lock lets two clients believe they hold the same lease, and how a fencing token catches it downstream.

text

T=0s    Client A: SET lock_key A_id NX PX 5000
                     -> acquires lease, TTL 5s, fencing token 42

T=1s    Client A starts a slow operation (unexpected
        GC pause / slow disk write) that takes 7s total

T=5s    Lease expires in the lock service — Client A
        is still mid-operation and does not know yet

T=5.1s  Client B: SET lock_key B_id NX PX 5000
                     -> succeeds (key was gone), fencing token 43

T=6s    Both A and B now believe they exclusively hold
        the lock. Without fencing tokens, both proceed
        to write the protected resource -> corruption.

        With fencing tokens: the protected resource
        remembers the highest token it has accepted (43,
        from B). When A's delayed write with token 42
        arrives at T=8s, the resource rejects it because
        42 < 43 -- the stale holder's write is caught
        even though the lock service itself was fooled.

1: The lease is acquired with a 5-second TTL and a fencing token of 42 — the token is the piece that survives the coming race.
7: The lease expiring is silent to Client A — nothing tells it in real time that it no longer holds the lock, which is the core of the stale-lock problem.
10: Client B successfully acquires the same lock key because it genuinely expired — both clients now believe, correctly by the lock service's own bookkeeping, that they hold it.
17: The fencing-token check happens at the protected resource, not at the lock service — that is what makes it work even after the lock service has already been fooled into granting two holders.

Why this works: This sequence is the textbook failure the roadmap item is pointing at: a lease alone bounds how long a stale holder can block others, but it does not by itself prevent two holders from overlapping — only a check at the protected resource (the fencing token) closes that gap.

Relying on the lease TTL alone as proof of exclusivity, with no fencing token

Wrong

text

if redis.set('lock', my_id, nx=True, px=5000):
    write_to_shared_file(payload)  # no token check
    redis.delete('lock')

Better

text

token = redis.set('lock', my_id, nx=True, px=5000, get=True)
fencing_token = get_next_fencing_token()
if token:
    write_to_shared_file(payload, fencing_token=fencing_token)
    # shared_file storage itself rejects any write whose
    # fencing_token is lower than the last one it accepted
    redis.delete('lock')

What you see: Data corruption or duplicate side effects that show up only under load or after a GC pause / slow network blip — the lock code looks correct in isolation and passes ordinary tests, because the failure needs a holder to genuinely overrun its lease to manifest.

Why: A TTL only bounds how long a lock can be held without being provably released — it says nothing about whether the original holder's in-flight write can still land after a second holder has already started; only a check that the protected resource itself performs (the fencing token) can catch a write arriving late from a client that no longer legitimately owns the lock.

A stale lock, and the fencing token that catches it

Client A

Lock service

Client B

Resource

1. acquire, TTL 5stoken 42
2. slow op (7s)GC pause — lease will outlive this
3. lease expirest=5s, A still working
4. acquire, TTL 5stoken 43 — succeeds
5. write, token 42rejected — 42 < 43 already accepted

Four risk categories in lease-based distributed locks

Four risk categories in lease-based distributed locks
Risk	What goes wrong	Mitigation
Stale lock	Holder still working past lease expiry; a second client acquires the same lock	Fencing tokens; keep lease TTL comfortably larger than expected work time; lock-extension heartbeats
Process crash	Holder dies without releasing; without a lease, the lock is held forever	Always use an expiring lease, never an unbounded lock
Lease expiration timing	TTL is a guess about work duration, not a measurement of it	Extend the lease from within the held work via a renewal/heartbeat call, rather than picking one large fixed TTL
Clock drift/wall-clock jumps	Expiry computed from wall-clock time can fire early or late relative to real elapsed time	NTP discipline on all hosts; treat lease validity as TTL minus a clock-drift margin, per Redis's own Redlock safety argument

Remember: A lease bounds how long a crashed holder blocks others, but a lease alone does not prevent overlap if the holder overruns it — that needs a fencing token checked at the protected resource. Base TTLs on measured work duration plus a clock-drift margin, not a guess.

The simpler alternative

Why a database's own uniqueness or atomic-operation guarantees usually make a distributed lock unnecessary.

Prefer database uniqueness or atomic operations over a lock

standardintermediate

Most invariants that sound like "we need a distributed lock" are actually one of two simpler shapes in disguise: "exactly one of these can exist" (a unique constraint) or "update this value only if it hasn't changed" (an atomic compare-and-set / atomic UPDATE). Both are enforced by the database itself, in a single round trip, with no separate lock to acquire, hold, extend, or risk leaving stale. Reach for an actual distributed lock only for the remaining case that constraint and atomic-operation approaches genuinely cannot express — coordinating a critical section that spans multiple calls or systems over time.

Think of it as

A unique constraint or atomic UPDATE is a bouncer built into the door itself — it only lets one person through, permanently, with no separate coordination needed. A distributed lock is hiring a separate security guard to stand at the door and radio other guards before letting anyone in. The guard works, but needs to be paid, kept awake, and trusted not to wander off — the built-in door is simpler whenever the invariant can be expressed as a single "let exactly one through" or "only update if unchanged" check at the door itself.

sql

-- claim exactly one pending job, atomically, no lock service
UPDATE jobs SET status = 'claimed', claimed_by = :worker_id
  WHERE id = :job_id AND status = 'pending'
  RETURNING id;
-- 1 row back: you claimed it. 0 rows: someone else did.

What we're doing: Replace a Redis-lock-guarded stock decrement with a single atomic UPDATE that expresses the same invariant.

atomic-stock-decrement.sqlsql

-- The invariant: never let stock go negative, even
-- under concurrent purchase requests.

-- Atomic compare-and-set expresses it directly:
UPDATE inventory
  SET stock = stock - 1
  WHERE product_id = 42 AND stock > 0
  RETURNING stock;

-- 1 row returned: this request's decrement succeeded
-- and stock stayed non-negative.
-- 0 rows returned: stock was already 0 — this request's
-- purchase attempt is rejected, no negative stock ever
-- existed even momentarily.

7: The WHERE stock > 0 clause is the entire concurrency guarantee — the database checks and updates in one atomic step, so two simultaneous requests can never both succeed past the last unit of stock.
12: A 0-row result is how the caller detects "someone else got the last one" — equivalent to what a distributed lock would have told a losing client, but without ever taking a lock.

Why this works: This is exactly the invariant a team might reach for a distributed lock to protect ("prevent overselling under concurrent requests"), and the atomic UPDATE enforces it with one query, no lease, and no risk of the stale-lock or clock-drift failure modes that a lock service introduces.

Wrapping a read-check-write in a distributed lock instead of pushing the check into the UPDATE itself

Wrong

text

lock = acquire_distributed_lock(f'stock:{product_id}')
if lock:
    stock = db.query('SELECT stock FROM inventory WHERE product_id = %s', product_id)
    if stock > 0:
        db.execute('UPDATE inventory SET stock = stock - 1 WHERE product_id = %s', product_id)
    release_distributed_lock(lock)

Better

text

row = db.execute(
    'UPDATE inventory SET stock = stock - 1 '
    'WHERE product_id = %s AND stock > 0 '
    'RETURNING stock', product_id)
sold = row is not None

What you see: A working feature that also now depends on a lock service being reachable and correctly configured for every single stock decrement — an outage or misconfiguration in that lock service turns into a purchasing outage, for an invariant the database could have enforced on its own the whole time.

Why: The read-check-write pattern only needs external locking because it splits one atomic idea (decrement, but not below zero) into three separate steps; folding the check into the WHERE clause makes the whole thing one atomic database operation again, which removes the need for external coordination entirely.

Match the invariant to the lightest mechanism

Unique constraint

exactly one of these

Atomic UPDATE...WHERE

only if unchanged

Claim via RETURNING

one worker claims the row

Distributed lock

multi-step critical section

Matching an invariant to the lightest mechanism that expresses it

Matching an invariant to the lightest mechanism that expresses it
Invariant	Lightest fit	Why a distributed lock is unnecessary here
"Only one row for this idempotency key"	Unique constraint	The database rejects the duplicate INSERT atomically — nothing to coordinate across processes
"Only decrement stock if it is still positive"	Atomic UPDATE ... WHERE stock > 0	One round trip; the WHERE clause is the entire check-and-act, done atomically by the database
"Only one worker claims this queued job"	UPDATE ... WHERE status = 'pending' RETURNING id	The row-level atomic update is itself the claim — no external lock service needed
"Only one instance of this multi-step batch job runs across the fleet, for its whole 10-minute duration"	Distributed lock (with a lease + fencing token)	The critical section spans multiple operations over real time — no single atomic database call covers "for the next 10 minutes, and only this process"

Remember: Before building a distributed lock, ask whether the invariant is really "exactly one" (unique constraint) or "only if unchanged" (atomic UPDATE ... WHERE) — both are enforced by the database in one round trip, with none of a lock's lease/clock-drift risk. Reach for a real distributed lock only when the critical section spans multiple operations over time that no single atomic call can cover.