beginnerintermediateadvanced

System Design · Distributed Locks

Distributed Locks

When coordination across processes is genuinely needed, and why a database's own guarantees are usually enough

When neededno single DB already owns the invariant
Lease riskstale locks, crashes, clock drift
Fencing tokensthe real fix, not a longer TTL
Prefer simplerunique constraint or atomic UPDATE

1.When distributed coordination is actually needed

Distributed coordination — a lock held across multiple processes or machines — is needed only when the resource being protected is itself outside any single database that could enforce the invariant directly. If one Postgres instance already holds the data and can express "only one of these" as a transaction or a constraint, that is not a distributed-locking problem at all; it becomes one when multiple independent workers must serialize access to something a single ACID transaction cannot already cover — a shared external resource, a scheduled job that must run on exactly one of many replicas, or a critical section spanning several services and calls.

Key features

If one transactional database already owns all the state involved, prefer its own transaction/constraint machinery over a distributed lock — it is simpler and already consistent
Distributed coordination becomes necessary when multiple independent processes (often on different machines) must serialize access to a resource no single database transaction spans
Common genuine use cases: leader election among replicas, ensuring a scheduled/cron job runs on exactly one worker, guarding a non-transactional external resource (a third-party API, a file on shared storage)
A distributed lock adds a new failure mode (the lock service itself, plus network partitions between clients and it) that a single-database transaction does not have
The question to ask first is not "how do I build the lock" but "does this invariant already live inside one system that can enforce it atomically" — most of the time it does

Deciding whether a problem needs a distributed lock

Deciding whether a problem needs a distributed lock
Situation	Coordination needed?	Why
Two web servers both writing to the same Postgres row	No	A transaction, unique constraint, or row lock inside Postgres already serializes this
Five worker instances running the same cron job, only one should execute it	Yes	No single database transaction spans "did any worker already claim this run" across independent process starts unless one is designated to own that check — a lock (or a leader) is the coordination mechanism
Multiple app servers calling a rate-limited third-party API that has no server-side idempotency	Yes	The invariant ("no more than N calls per window") lives outside any database the app controls
One microservice validating a request before writing to its own database	No	Single service, single database — ordinary transaction scope covers it

Does one database already own this invariant?

yesno

Shared invariant

multiple processes involved

One DB already owns it

use a transaction/constraint

Spans systems

needs a distributed lock

Signature

distributed lock needed ⇔ invariant spans systems no single DB transaction covers

Ask "does one database already own this invariant?" before reaching for a distributed lock — if yes, use its transaction/constraint machinery; a distributed lock is for coordination that genuinely spans systems no single database transaction covers.

2.Stale locks, crashes, lease expiration and clock assumptions

A distributed lock is almost always implemented as a lease — a key that auto-expires after a timeout, so the system does not deadlock forever if the holder crashes. That auto-expiry is also exactly where the risk lives: if the holder is still working when the lease expires (a long garbage-collection pause, a slow network, a scheduler delay), another client can acquire the same lock while the first one believes it still holds it — two processes now both think they have exclusive access at once. Redis's own Redlock documentation is explicit that mutual exclusion is only guaranteed if the holder finishes its work within the lease's validity time minus clock drift, and recommends fencing tokens as the real fix, not just a longer lease.

Key features

Stale lock: the holder is still doing work when the lease expires (GC pause, slow I/O, scheduler delay, or a slow network to the lock service) — a second client can then acquire the same lock while the first still believes it holds it
Process crash: if a holder crashes without releasing the lock, the lease's auto-expiry is what prevents a permanent deadlock — but a lease long enough to be safe from stale-holder races is also a lease that keeps the resource unavailable for that long after a real crash
Lease expiration is a timing bet, not a guarantee: it assumes the holder's actual work finishes comfortably inside the TTL; Redis's own docs frame the safety guarantee as holding only "as long as the client holding the lock terminates its work within the lock validity time... minus some time... to compensate for clock drift"
Clock assumptions: lease-based locks (including Redis's own Redlock) rely on every process's local clock advancing at roughly the same rate — a wall-clock jump (NTP correction, manual clock change) can make a lease expire early or late relative to what the holder believes, and Redis's docs flag this directly since Redis uses wall-clock time, not a monotonic clock, for TTL expiration
Fencing tokens are the standard mitigation: the lock service hands out a monotonically increasing number with each lock grant, and the protected resource itself rejects any write tagged with an older token than one it already accepted — this catches a stale holder even if the lock system itself briefly said two clients both "had" the lock

Four risk categories in lease-based distributed locks

Four risk categories in lease-based distributed locks
Risk	What goes wrong	Mitigation
Stale lock	Holder still working past lease expiry; a second client acquires the same lock	Fencing tokens; keep lease TTL comfortably larger than expected work time; lock-extension heartbeats
Process crash	Holder dies without releasing; without a lease, the lock is held forever	Always use an expiring lease, never an unbounded lock
Lease expiration timing	TTL is a guess about work duration, not a measurement of it	Extend the lease from within the held work via a renewal/heartbeat call, rather than picking one large fixed TTL
Clock drift/wall-clock jumps	Expiry computed from wall-clock time can fire early or late relative to real elapsed time	NTP discipline on all hosts; treat lease validity as TTL minus a clock-drift margin, per Redis's own Redlock safety argument

A stale lock, and the fencing token that catches it

Client A

Lock service

Client B

Resource

1acquire, TTL 5stoken 42
2slow op (7s)GC pause — lease will outlive this
3lease expirest=5s, A still working
4acquire, TTL 5stoken 43 — succeeds
5write, token 42rejected — 42 < 43 already accepted

Signature

lease TTL bounds crash recovery; fencing tokens (not TTL alone) prevent overlap

A lease bounds how long a crashed holder blocks others, but a lease alone does not prevent overlap if the holder overruns it — that needs a fencing token checked at the protected resource. Base TTLs on measured work duration plus a clock-drift margin, not a guess.

3.Prefer database uniqueness or atomic operations over a lock

Most invariants that sound like "we need a distributed lock" are actually one of two simpler shapes in disguise: "exactly one of these can exist" (a unique constraint) or "update this value only if it hasn't changed" (an atomic compare-and-set / atomic UPDATE). Both are enforced by the database itself, in a single round trip, with no separate lock to acquire, hold, extend, or risk leaving stale. Reach for an actual distributed lock only for the remaining case that constraint and atomic-operation approaches genuinely cannot express — coordinating a critical section that spans multiple calls or systems over time.

Key features

A unique constraint (or a unique partial index) enforces "exactly one of these" atomically, in the same operation that creates the row — no lock acquire/release cycle at all
An atomic UPDATE ... WHERE (compare-and-set) enforces "only change this if it still matches what I expect" in one round trip — see system-design.isolation-levels-and-concurrency.concurrency-control-mechanisms for the general mechanism
These database-native mechanisms have no lease, no TTL, no clock-drift assumption, and no separate service to keep available — the risks specific to distributed locks (stale locks, lease expiration timing) do not apply to them at all
A genuine distributed lock is still the right tool when the critical section spans multiple operations or systems over a stretch of real time — e.g. "only one process may run this multi-step job right now" — which a single atomic database operation cannot express
The decision is not "locks are bad" — it is "check whether the invariant fits in one atomic database operation before building a lock service to protect a multi-step critical section"

Matching an invariant to the lightest mechanism that expresses it

Matching an invariant to the lightest mechanism that expresses it
Invariant	Lightest fit	Why a distributed lock is unnecessary here
"Only one row for this idempotency key"	Unique constraint	The database rejects the duplicate INSERT atomically — nothing to coordinate across processes
"Only decrement stock if it is still positive"	Atomic UPDATE ... WHERE stock > 0	One round trip; the WHERE clause is the entire check-and-act, done atomically by the database
"Only one worker claims this queued job"	UPDATE ... WHERE status = 'pending' RETURNING id	The row-level atomic update is itself the claim — no external lock service needed
"Only one instance of this multi-step batch job runs across the fleet, for its whole 10-minute duration"	Distributed lock (with a lease + fencing token)	The critical section spans multiple operations over real time — no single atomic database call covers "for the next 10 minutes, and only this process"

Match the invariant to the lightest mechanism

Unique constraint

exactly one of these

Atomic UPDATE...WHERE

only if unchanged

Claim via RETURNING

one worker claims the row

Distributed lock

multi-step critical section

Command

sql

-- claim exactly one pending job, atomically, no lock service
UPDATE jobs SET status = 'claimed', claimed_by = :worker_id
  WHERE id = :job_id AND status = 'pending'
  RETURNING id;
-- 1 row back: you claimed it. 0 rows: someone else did.

Before building a distributed lock, ask whether the invariant is really "exactly one" (unique constraint) or "only if unchanged" (atomic UPDATE ... WHERE) — both are enforced by the database in one round trip, with none of a lock's lease/clock-drift risk. Reach for a real distributed lock only when the critical section spans multiple operations over time that no single atomic call can cover.

Spotted a mistake on this page? Tell us.

Read Sheet