Leader Election and Coordination

How one node proves it is the leader, how failover can go wrong, and what ZooKeeper/etcd solve

Heartbeats & leaseshow a leader proves it is still alive
Split-braintwo nodes, one partition, both "leader"
ZooKeeper / etcda consistent place to store "who is leader"
ConsensusRaft / ZAB — majority agreement, no split-brain

1.Leader/follower roles, heartbeats and leases

In a leader/follower design, exactly one node (the leader) is allowed to make a certain class of decisions — accept writes, assign work, or order events — while the rest (followers) accept the leader's decisions instead of racing to make their own. Because the leader can crash or lose its network connection at any moment, the system needs a way to keep proving "the current leader is still alive and in charge": a heartbeat is the leader (or a health checker) repeatedly signaling aliveness, and a lease is a time-bounded grant of leadership that automatically expires unless renewed — so followers know exactly how long they can trust the current leader before it is safe to consider the role vacant.

Key features

A heartbeat is a periodic "I'm still alive" signal; followers (or a monitor) infer leader failure only after missing some number of expected heartbeats within a timeout window.
A lease is a time-bounded grant: "you are leader until time T," which expires automatically if not renewed — the leader must actively renew it before T to keep the role.
Leases turn "did the leader crash?" (hard to know for certain) into "has the lease expired?" (a simple clock check) — this is why leases, not raw heartbeats alone, are the safer building block for leader-only actions.
A leader that assumes it still holds the lease past its expiry (e.g. paused by a long garbage-collection pause) can keep acting as leader even though everyone else has already elected a new one — clock and pause assumptions matter, not just the lease mechanism itself.
Follower roles are typically passive: replicate the leader's log/state and be ready to take over, rather than independently deciding anything the leader is responsible for.

Heartbeat vs lease, as an aliveness mechanism

Heartbeat vs lease, as an aliveness mechanism
Property	Heartbeat alone	Lease
What proves aliveness	Recent signal received	An unexpired time grant
Failure detection	Missed N heartbeats within a timeout	Lease expiry time passes with no renewal
Risk if clocks/pauses are ignored	False "still alive" if signal is merely delayed	False belief of still holding leadership past a paused renewal
Typical use	General liveness checks between any nodes	Leader election specifically — one exclusive grant at a time

Lease renewal, a crash, and the timeout that triggers a new election

Node A

Followers

Node B

1heartbeat, renewslease expires t=13s
2heartbeat, renewslease expires t=16s
3crashest=9s — followers don't know yet
4lease expires, t=16sno renewal received
5new electionB wins, lease until t=27s

Command

text

leader:    holds the lease, sends heartbeats,
           renews the lease before it expires
follower:  watches for heartbeats / lease expiry,
           replicates leader state, stands by
lease:     "leader until time T" — expires
           automatically unless renewed before T

A heartbeat proves recent aliveness; a lease proves aliveness for a bounded, expiring window — leases are safer for leader-only actions because "has it expired?" is a simple clock check, not an inference from a missed signal.

2.Failover and the split-brain risk

Failover is the process of promoting a follower to leader after the old leader is believed to have failed, so the system keeps working without a human manually intervening. The dangerous failure mode failover can trigger is split-brain: the old leader has not actually stopped (it was only unreachable, e.g. a network partition, a long pause), so once a new leader is promoted there are briefly two nodes that both believe they alone are leader — and both may accept writes, producing two diverging, conflicting histories of the same data.

Key features

Split-brain's trigger condition is a network partition (or a long pause) that makes a healthy leader unreachable to followers, not an actual crash of the leader process.
The concrete danger is two leaders both accepting writes during the overlap window, producing conflicting data that must be reconciled or is silently lost.
Fencing (forcibly cutting an old leader's ability to write — e.g. revoking its storage access or shutting off its network port) is the standard defense used alongside failover, not a replacement for it.
A quorum requirement (a new leader needs votes from a majority of nodes, not just any one node) prevents two leaders from being elected on each side of a partition — a minority side can never elect its own leader.
Fast automatic failover reduces downtime but increases split-brain risk if promotion happens before the old leader's lease has genuinely expired; conservative failover (waiting out a full lease timeout) trades speed for safety.

A minority side cannot win an election, by design

A minority side cannot win an election, by design
Scenario	Nodes reachable	Majority of 5?	Can elect a leader?
No partition	All 5	Yes (5/5)	Yes
Partition, majority side	3 of 5	Yes (3/5)	Yes
Partition, minority side	2 of 5	No (2/5)	No — stays leaderless until reconnected

Split-brain: A is partitioned, not dead

Node A

Node B

Node C

1network partitionA still running, unreachable
2writes balance=100client X, still connected to A
3lease expires, electB wins, becomes leader
4writes balance=200client Y, connected to B
5partition heals, replicates 100100 vs 200 — both accepted

Command

text

failover:    promote a follower to leader after
             the old leader is believed failed
split-brain: two nodes both believe they are
             leader at the same time
fencing:     forcibly block the old leader from
             writing once it's been replaced

Split-brain's root cause is a leader that is unreachable, not dead — failover promotes a new leader while the old one may still be accepting writes; fencing (cut the old leader off) and quorum (a minority partition can never elect a leader) are the actual defenses, not a longer timeout alone.

3.Coordination systems: ZooKeeper, etcd and consensus

A coordination system (ZooKeeper, etcd, and similar) is a small, separately-run cluster whose one job is to be a highly reliable, consistent place to store a tiny amount of shared state that every other node in a distributed system needs to agree on — most commonly "who is the leader right now." Instead of every application re-inventing leader election, heartbeats, leases and split-brain-safe quorum logic from scratch, applications delegate that hard problem to a system that has already solved it once, correctly, using a consensus protocol (like Raft, which etcd uses, or ZAB, which ZooKeeper uses) that itself tolerates node failures without losing agreement.

Key features

etcd's own docs describe it as "a consistent key-value store for configuration management, service discovery, and coordinating distributed work," explicitly designed for systems that "will never tolerate split-brain operation" — this is the exact split-brain risk from this section's failover concept, solved once at the coordination layer.
ZooKeeper's own docs describe it as "a distributed, open-source coordination service for distributed applications," built because "coordination services are notoriously hard to get right" — it exists to relieve every application of implementing leader election, locks and config sync from scratch.
Both etcd (via Raft) and ZooKeeper (via ZAB, its own atomic broadcast protocol) use a consensus protocol that requires a majority of the coordination cluster's own nodes to agree before any state change is considered committed.
A common pattern built on top of these systems is: a node acquires a lease/ephemeral node representing "leader," other nodes watch for it to disappear (crash, expiry, disconnect), and whichever node next acquires it becomes the new leader — the primitives from this section's first two concepts, implemented once, correctly, by the coordination system.
Kubernetes' own control plane is a concrete real-world example: its API server persists all cluster state into etcd and uses etcd's watch API to detect and react to changes — a coordination system underpinning a widely-used production system, not just a theoretical building block.

What a coordination system replaces, and what it costs

What a coordination system replaces, and what it costs
Aspect	Roll your own	Delegate to ZooKeeper/etcd
Consensus protocol correctness	Must implement and prove correct yourself	Already implemented (Raft/ZAB) and battle-tested
Split-brain prevention	Must design quorum + fencing yourself	Built in — the coordination cluster itself never splits its own view of the leader key
Operational cost	None beyond your own service	An additional clustered system to run, monitor and upgrade
Failure domain	Coupled to your service's own nodes	A separate, dedicated cluster — a single well-tested dependency shared by many services

Application nodes delegate "who is leader?" to one cluster

Application nodes

Node A

Node B

Node C

Coordination cluster

Exactly one winner

Command

text

application nodes  --watch/lease-->  coordination
   (many, varied)                        cluster
                                    (ZooKeeper / etcd,
                                     odd # of nodes,
                                     consensus-based)
"who is leader?" is answered by the coordination
cluster's own agreed-upon state, not by any one
application node's local belief.

ZooKeeper and etcd exist so applications don't reinvent split-brain-safe leader election themselves — they provide one consistent, fault-tolerant place to store "who is the leader," backed by a consensus protocol (ZAB, Raft) that requires a majority of the coordination cluster's own nodes to agree, so run them as an odd-sized cluster, not a single node.

Spotted a mistake on this page? Tell us.

Read Sheet