beginnerintermediateadvanced

System Design · Section 43

Leader Election and Coordination

Level: intermediate
Read: 15 min
Concepts: 3

Leader/follower designs let exactly one node make a class of decisions while the rest stand by — heartbeats give a periodic aliveness signal, and leases give a safer, time-bounded grant of leadership that expires automatically unless renewed. Failover promotes a follower to leader once the old leader is believed gone, but if the old leader was only unreachable (a network partition) rather than actually dead, both can end up believing they are leader at once — split-brain, the worked failure example this section walks through. Coordination systems like ZooKeeper and etcd exist to solve this once, correctly: a small, consensus-backed cluster (Raft, ZAB) that gives every application a single consistent, fault-tolerant place to store "who is the leader," so individual services don't reimplement split-brain-safe election themselves.

This section

Implementing Leader Election / Distributed lock In Distributed Systems using ZooKeeper-System DesignBalasubramanian Bangalore

What is true here

A lease ("leader until time T", expiring unless renewed) is a safer aliveness proof than a bare heartbeat, because expiry is a clock check rather than an inference from a missed signal.
Split-brain's trigger is an unreachable-but-still-running leader, not a crashed one — the old leader keeps acting normally while a new one is elected around it.
Fencing (blocking the old leader's writes) and quorum (a minority partition can never elect a leader) are the actual defenses against split-brain, not a longer failover timeout.
ZooKeeper (ZAB) and etcd (Raft) provide a consensus-backed, split-brain-safe place to store leadership state, so applications delegate leader election instead of reimplementing it — but the coordination cluster itself must be run as an odd-sized cluster to avoid becoming a new single point of failure.

What you will be able to do

Explain how a lease differs from a plain heartbeat and why it is the safer building block for leader-only actions
Walk through a concrete split-brain scenario and identify exactly where the conflicting writes were accepted
Name fencing and quorum as the actual defenses against split-brain, not just a longer timeout
Explain, conceptually, what problem ZooKeeper/etcd solve and why the coordination cluster itself needs an odd-sized majority

Leader/follower roles and proving aliveness

How a leader proves to the rest of the system that it is still alive and in charge.

Leader/follower roles, heartbeats and leases

coreintermediate

In a leader/follower design, exactly one node (the leader) is allowed to make a certain class of decisions — accept writes, assign work, or order events — while the rest (followers) accept the leader's decisions instead of racing to make their own. Because the leader can crash or lose its network connection at any moment, the system needs a way to keep proving "the current leader is still alive and in charge": a heartbeat is the leader (or a health checker) repeatedly signaling aliveness, and a lease is a time-bounded grant of leadership that automatically expires unless renewed — so followers know exactly how long they can trust the current leader before it is safe to consider the role vacant.

Think of it as

A lease is like a hotel key card programmed to stop working at checkout time. The guest (leader) doesn't have to hand the card back for the hotel to know the room is available again — the card simply stops opening the door once the clock passes checkout, unless the guest walks to the front desk and renews it first. Nobody has to detect that the guest left; the expiry does the work. A heartbeat is the simpler version: the guest calling the front desk every few minutes just to say "still here" — if the calls stop, the front desk assumes the room is free after a timeout, even though it can't be fully sure the guest didn't just lose phone signal.

text

leader:    holds the lease, sends heartbeats,
           renews the lease before it expires
follower:  watches for heartbeats / lease expiry,
           replicates leader state, stands by
lease:     "leader until time T" — expires
           automatically unless renewed before T

What we're doing: Trace one lease renewal cycle and the timeout that would trigger a new election.

lease-cycle.txttext

t=0s    Node A wins election, granted lease until t=10s
t=3s    Node A sends heartbeat, renews lease -> expires t=13s
t=6s    Node A sends heartbeat, renews lease -> expires t=16s
t=9s    Node A crashes (process dies)
t=16s   Lease expires with no renewal received
t=16s   Followers detect expiry, start a new election
t=17s   Node B wins, granted lease until t=27s

2: The lease is granted with a hard expiry time, not an indefinite claim — this is what makes "leader is gone" detectable by a clock instead of a guess.
6: The crash happens between renewals; followers have no way to know yet — they can only wait for the already-granted lease to run out.
7: Only once the lease expires (not the moment of the crash) do followers treat the role as vacant and start a new election — a deliberate delay that trades detection speed for safety.

Why this works: This shows the actual timing gap every lease-based design accepts: there is always a window between a real failure and the system noticing, sized by the lease duration — a shorter lease detects failure faster but demands more frequent renewal traffic and network overhead.

Treating "missed one heartbeat" as proof the leader is dead

Wrong

text

if (no heartbeat received this tick) {
  startNewElection();
}

Better

text

if (now > lease.expiresAt) {
  // lease genuinely expired, not just one
  // slow/lost heartbeat message
  startNewElection();
}

What you see: Elections trigger constantly under normal network jitter or GC pauses, with leadership flapping between nodes even though no node has actually failed.

Why: A single missed heartbeat is common under ordinary network delay or a garbage-collection pause and does not mean the leader is down — a lease with a deliberate expiry window (built from multiple missed intervals, not one) avoids triggering elections on transient noise.

Lease renewal, a crash, and the timeout that triggers a new election

Node A

Followers

Node B

1. heartbeat, renewslease expires t=13s
2. heartbeat, renewslease expires t=16s
3. crashest=9s — followers don't know yet
4. lease expires, t=16sno renewal received
5. new electionB wins, lease until t=27s

Heartbeat vs lease, as an aliveness mechanism

Heartbeat vs lease, as an aliveness mechanism
Property	Heartbeat alone	Lease
What proves aliveness	Recent signal received	An unexpired time grant
Failure detection	Missed N heartbeats within a timeout	Lease expiry time passes with no renewal
Risk if clocks/pauses are ignored	False "still alive" if signal is merely delayed	False belief of still holding leadership past a paused renewal
Typical use	General liveness checks between any nodes	Leader election specifically — one exclusive grant at a time

Remember: A heartbeat proves recent aliveness; a lease proves aliveness for a bounded, expiring window — leases are safer for leader-only actions because "has it expired?" is a simple clock check, not an inference from a missed signal.

Failover and its split-brain risk

What can go wrong when a follower is promoted while the old leader is still running, just unreachable.

Failover and the split-brain risk

coreintermediate

Failover is the process of promoting a follower to leader after the old leader is believed to have failed, so the system keeps working without a human manually intervening. The dangerous failure mode failover can trigger is split-brain: the old leader has not actually stopped (it was only unreachable, e.g. a network partition, a long pause), so once a new leader is promoted there are briefly two nodes that both believe they alone are leader — and both may accept writes, producing two diverging, conflicting histories of the same data.

Think of it as

Split-brain is like a company where the CEO steps out of contact on a delayed flight, and the board — unable to reach them — appoints a new CEO to keep things running. The catch: the old CEO's phone was just off, not the CEO gone. They land, turn their phone back on, and start signing contracts again, not knowing they were replaced. Now two people are signing contracts as "the CEO" for the same company at the same time, and the contracts disagree.

text

failover:    promote a follower to leader after
             the old leader is believed failed
split-brain: two nodes both believe they are
             leader at the same time
fencing:     forcibly block the old leader from
             writing once it's been replaced

What we're doing: Walk through a concrete split-brain: a network partition isolates the leader, a new leader is elected, and both accept writes.

split-brain.txttext

t=0s   Node A is leader; B and C are followers.
       A, B, C can all reach each other.
t=5s   Network partition: A can no longer reach
       B or C (A is still running fine).
t=5s   Client X, still connected to A, writes
       balance = 100 to A.
t=15s  B and C stop hearing A's heartbeats,
       A's lease (granted at t=0, 10s window)
       has now expired.
t=15s  B and C hold an election; B wins,
       becomes the new leader.
t=16s  Client Y, connected to B, writes
       balance = 200 to B.
t=20s  Network partition heals; A reconnects.
t=20s  A still believes it is leader (it never
       saw the election) and now replicates its
       balance = 100 to B and C.
       -- Two conflicting values (100 vs 200)
       -- were both accepted as "the" balance.

4: The partition isolates A without crashing it — this is the precondition for split-brain, not a leader crash.
6: A keeps behaving exactly as a normal leader would, including accepting writes, because from A's point of view nothing is wrong.
15: B is elected only after A's lease genuinely expires, not merely after missed heartbeats — yet split-brain still happens, because A is still alive and still holds client X's write.
14: The conflict surfaces once the partition heals and A tries to act as leader again — two writes were accepted by two different "leaders" for the same logical value.

Why this works: This is the scenario the term "split-brain" specifically names — not two leaders started by mistake, but one still-legitimate-feeling leader and one newly-elected leader, both correct from their own local point of view, disagreeing about reality.

Believing a longer heartbeat timeout alone prevents split-brain

Wrong

text

"We'll set the failover timeout to 60 seconds
instead of 10 — that gives plenty of time to
avoid electing a new leader by mistake."

Better

text

"A longer timeout reduces how often we
false-trigger failover on transient network
blips, but it doesn't prevent split-brain during
a real, sustained partition — we still need
fencing (block the old leader's writes once
replaced) and a quorum requirement (an isolated
minority can't elect its own leader) as the
actual safety mechanisms."

What you see: Two nodes both serve writes as "the leader" for the duration of any partition that outlasts the timeout, regardless of how long that timeout is set to.

Why: A timeout only controls how fast failover happens, not whether split-brain can happen during a real partition — a longer timeout just makes false failovers rarer while doing nothing to stop a genuine, sustained partition from producing two leaders.

Split-brain: A is partitioned, not dead

Node A

Node B

Node C

1. network partitionA still running, unreachable
2. writes balance=100client X, still connected to A
3. lease expires, electB wins, becomes leader
4. writes balance=200client Y, connected to B
5. partition heals, replicates 100100 vs 200 — both accepted

A minority side cannot win an election, by design

A minority side cannot win an election, by design
Scenario	Nodes reachable	Majority of 5?	Can elect a leader?
No partition	All 5	Yes (5/5)	Yes
Partition, majority side	3 of 5	Yes (3/5)	Yes
Partition, minority side	2 of 5	No (2/5)	No — stays leaderless until reconnected

Remember: Split-brain's root cause is a leader that is unreachable, not dead — failover promotes a new leader while the old one may still be accepting writes; fencing (cut the old leader off) and quorum (a minority partition can never elect a leader) are the actual defenses, not a longer timeout alone.

Coordination systems, conceptually

Why ZooKeeper, etcd and consensus-based systems exist, and what problem they solve for the scenario above.

Coordination systems: ZooKeeper, etcd and consensus

coreintermediate

A coordination system (ZooKeeper, etcd, and similar) is a small, separately-run cluster whose one job is to be a highly reliable, consistent place to store a tiny amount of shared state that every other node in a distributed system needs to agree on — most commonly "who is the leader right now." Instead of every application re-inventing leader election, heartbeats, leases and split-brain-safe quorum logic from scratch, applications delegate that hard problem to a system that has already solved it once, correctly, using a consensus protocol (like Raft, which etcd uses, or ZAB, which ZooKeeper uses) that itself tolerates node failures without losing agreement.

Think of it as

A coordination system is like a courthouse's single, official land registry office, instead of every neighbor keeping their own notebook of who owns which plot. Neighbors could each track ownership themselves, but their notebooks would disagree the moment two neighbors both wrote down conflicting claims during a dispute. The registry office exists so there is exactly one authoritative, durable answer to "who owns this plot right now" — and the office itself is built (multiple clerks, strict procedures, majority sign-off before any record changes) so that even if one clerk is unreachable, the registry as a whole still gives a single consistent answer instead of splitting into two rival records.

text

application nodes  --watch/lease-->  coordination
   (many, varied)                        cluster
                                    (ZooKeeper / etcd,
                                     odd # of nodes,
                                     consensus-based)
"who is leader?" is answered by the coordination
cluster's own agreed-upon state, not by any one
application node's local belief.

What we're doing: Show the leader-election pattern applications build on top of a coordination system, using etcd-style leases.

coordination-leader-election.txttext

1. Node A, B, C all start up and each try to
   create the key "/leader" with a lease attached.
2. etcd's consensus (Raft) guarantees only ONE of
   these create calls actually succeeds, even
   though all three were sent around the same time.
3. Node A's create succeeds; A is now leader.
   A must periodically renew its lease on "/leader"
   to keep the key alive.
4. B and C set a watch on "/leader" and do nothing
   else — they wait for a notification.
5. A crashes; A stops renewing its lease.
6. etcd expires the lease; the "/leader" key is
   deleted automatically by etcd itself.
7. etcd notifies B and C (the watch fires).
8. B and C race to create "/leader" again;
   etcd's consensus again picks exactly one winner.

2: The "only one create succeeds" guarantee is the whole point — etcd's Raft consensus resolves the race across three simultaneous attempts into a single winner, which is exactly the hard problem a hand-rolled system would have to solve itself.
6: etcd expires and deletes the key itself, based on the lease etcd is tracking — the application nodes do not need their own failure-detection logic; they only need to watch one key.
9: The same consensus guarantee from step 2 applies again here — even with B and C racing at the same moment, exactly one of them becomes the new leader, with no split-brain window at the coordination layer itself.

Why this works: This is the concrete mechanism that makes "delegate leader election to ZooKeeper/etcd" more than a buzzword — the application logic shrinks to "try to create one key, watch it if you fail," while the hard consensus problem (exactly one winner, safe under concurrent attempts and node failures) is solved once, inside the coordination cluster.

Assuming the coordination cluster itself can never be a single point of failure

Wrong

text

"We'll run one etcd node — it's simpler to
operate and it's still 'using etcd' for leader
election."

Better

text

"We need an odd-sized cluster of etcd nodes
(typically 3 or 5) so a majority can still form
and make progress if one node fails — a single
etcd node is itself a single point of failure,
which defeats the reason we adopted it."

What you see: Leader election for the whole application halts entirely the moment the one coordination node is restarted, deployed, or fails — a coordination outage now takes down every service that depends on it.

Why: Consensus protocols like Raft and ZAB need a majority of their own cluster members to agree — with one node, "a majority" is trivially satisfied until that one node is gone, at which point there is no coordination service left at all; the reliability the pattern promises only holds with a properly sized, odd-numbered cluster.

Application nodes delegate "who is leader?" to one cluster

Application nodes

Node A

Node B

Node C

Coordination cluster

Exactly one winner

What a coordination system replaces, and what it costs

What a coordination system replaces, and what it costs
Aspect	Roll your own	Delegate to ZooKeeper/etcd
Consensus protocol correctness	Must implement and prove correct yourself	Already implemented (Raft/ZAB) and battle-tested
Split-brain prevention	Must design quorum + fencing yourself	Built in — the coordination cluster itself never splits its own view of the leader key
Operational cost	None beyond your own service	An additional clustered system to run, monitor and upgrade
Failure domain	Coupled to your service's own nodes	A separate, dedicated cluster — a single well-tested dependency shared by many services

Remember: ZooKeeper and etcd exist so applications don't reinvent split-brain-safe leader election themselves — they provide one consistent, fault-tolerant place to store "who is the leader," backed by a consensus protocol (ZAB, Raft) that requires a majority of the coordination cluster's own nodes to agree, so run them as an odd-sized cluster, not a single node.