beginnerintermediateadvanced

System Design · Section 41

CAP Theorem

Level: intermediate
Read: 15 min
Concepts: 3

The CAP theorem describes one specific moment: a network partition has split a distributed data system, and a request needs data from the unreachable side. At that moment, the system must choose between Consistency (refuse/delay until certain) and Availability (answer locally, possibly stale) for the affected operations — it cannot have both for them. This is commonly misread as "every system permanently picks two of three properties," but partition tolerance is not realistically optional and most systems are both consistent and available outside of an actual partition. Real systems make opposite defaults for the same scenario: etcd (Raft, needs quorum) rejects writes on a minority partition (CP); Dynamo-style stores accept writes on every reachable side and reconcile conflicts afterward (AP).

This section

2. CAP Theorem (Hindi) | High Level Design for Beginners | CAP Partition Tolerance explainedConcept && Coding - by Shrayansh

What is true here

CAP applies during an actual network partition, for the operations that need the unreachable side — not as a permanent, system-wide setting.
The choice is Consistency (correctness first, may return errors) vs. Availability (uptime first, may return stale data) for those operations.
"Pick 2 of 3" is a common misreading — partition tolerance is not realistically optional, so most systems are both C and A outside of a partition.
Real systems differ concretely: etcd/Raft rejects minority-side writes (CP, fails loud); Dynamo-style stores accept writes everywhere and reconcile later (AP, fails quiet).

What you will be able to do

State precisely what CAP does and does not constrain, without the "pick 2 permanently" misreading
Trace a concrete split-brain scenario through both a CP and an AP resolution
Explain why a Raft-based system like etcd defaults to CP and what that costs during a partition
Explain why a Dynamo-style store defaults to AP and what reconciliation cost that shifts to later
Judge, for a specific piece of data, whether a CP or AP default is the safer choice

What CAP actually constrains

The precise partition-time choice CAP describes, and the "pick 2 of 3 permanently" misreading corrected.

What CAP actually says

coreintermediate

The CAP theorem describes one specific moment: a network partition has split a distributed data system into two (or more) groups of nodes that cannot talk to each other, and a request arrives at a node that would need to coordinate with the unreachable side to answer safely. At that exact moment, the system has only two choices for that request — answer it anyway using only local, possibly-stale data (favoring Availability), or refuse to answer until the partition heals and the nodes can agree (favoring Consistency). It cannot do both, because doing both would require talking to a node it cannot reach.

Think of it as

Picture two bank branches of the same bank that lose their phone line to head office and to each other, mid-afternoon, each still open for business. A customer walks into Branch A and asks to withdraw the full balance of a joint account. Branch A has no way to confirm whether Branch B already paid out the same balance to the other account holder an hour earlier. Branch A has exactly two honest options: refuse the withdrawal until the phone line is back and the branches can compare records (consistency wins, the customer is turned away), or pay out anyway on the balance Branch A last knew about (availability wins, but the bank now risks having paid out the same money twice). There is no third option where Branch A both stays open and guarantees it is not being fooled by stale local records — that is exactly the choice CAP describes.

text

Partition detected -> request needs cross-partition data
  -> choose C: return error/timeout until reachable
  -> choose A: return local (possibly stale) answer

What we're doing: Walk through a concrete split-brain moment and the two possible outcomes for the same write.

split-brain.txttext

Cluster: nodes A and B replicate the same key "cart:42"
Network partition: A and B can no longer reach each other
Both A and B are still up and still serving client requests

Client 1 -> talks to A -> writes cart:42 = {qty: 1}
Client 2 -> talks to B -> writes cart:42 = {qty: 3}

If the system favors Consistency (CP):
  - Neither A nor B can confirm it holds the latest value
  - One or both writes are rejected with an error/timeout
  - No conflicting values are ever visible to clients

If the system favors Availability (AP):
  - Both writes are accepted locally, on A and on B
  - cart:42 now disagrees between A and B
  - When the partition heals, the system must reconcile
    the two versions (e.g. last-write-wins, merge, or an
    application-level resolution)

3: This is the partition itself — both nodes are alive and reachable by clients, just not by each other.
6: Two clients send conflicting writes to the same logical key, one to each side of the split.
15: The AP choice accepts both writes, which is exactly what creates the conflict that must be reconciled later.

Why this works: CAP is easiest to misjudge in the abstract — seeing the same key diverge on two live nodes at once makes concrete why "just do both" is not on the table for that specific write.

Assuming a partition means a node is down

Wrong

text

"If a node can't reach the others, it's just
crashed — the crashed node's downtime is the
whole story."

Better

text

"A partition means the node is alive and still
serving clients — it just can't reach its peers.
That's what makes CAP a real choice: a genuinely
crashed node can't answer at all, but a partitioned
node actively chooses whether to answer using
only local information."

What you see: A postmortem describes a node as "down" when logs show it was actually accepting and answering client requests the whole time — it had simply lost its link to the rest of the cluster, which is a materially different failure with a materially different fix.

Why: The CAP theorem only has teeth because the partitioned node is still alive and capable of answering — a truly crashed node is just unavailable, which is a simpler and less interesting case than the live-but-isolated split brain CAP is actually about.

Split-brain: two writes, one key, a broken link

Client 1

Node A

Node B

Client 2

1. network partitionA and B can no longer reach each other
2. write cart:42 = {qty: 1}
3. write cart:42 = {qty: 3}
4. AP: both accepteddiverges — must reconcile once healed

The two choices available during an actual partition

The two choices available during an actual partition
Choice	What the node does	Cost
Favor Consistency (C)	Refuses or delays the request until it can confirm the latest state	Reduced availability — some requests get errors or timeouts
Favor Availability (A)	Answers using local data, without confirming it against the unreachable side	Reduced consistency — the answer may be stale or later conflict with the other side

Remember: CAP is a statement about one moment: a network partition plus a conflicting operation. At that moment, favor Consistency (refuse/delay until certain) or Availability (answer locally, possibly stale) — not both, for that operation.

CAP is not "pick 2 of 3, permanently"

standardintermediate

The popular shorthand "pick two of Consistency, Availability, Partition tolerance" is a common misreading. Partition tolerance is not something most real systems can opt out of — networks fail regardless of what a system "picks" — so it is not a peer choice alongside C and A. What CAP actually constrains is narrower: only while a partition is actually happening, and only for the specific operations that would need to coordinate across the split, does a system have to give up either consistency or availability. Outside of a partition, the same system is normally both consistent and available at once — there is no theorem forcing it to sacrifice one permanently.

Think of it as

It is like saying a car "must choose between driving and steering" — true only at the exact moment a wheel comes off, and only for that wheel. The rest of the time the car does both perfectly well. CAP's trade-off is a statement about behavior during a specific failure condition, not a permanent design constraint the car (or the system) lives under for its entire lifetime.

text

No partition:  system is both consistent AND available
Partition, operation needs the other side:
  -> now, and only now, choose C or A for that operation

What we're doing: Show the same system behaving as both C and A in normal operation, and only facing the CAP trade-off once a partition actually starts.

not-permanent.txttext

Timeline for a single-region, multi-node cluster:

09:00  Network healthy. Every write replicates to all
       nodes before acknowledging. Every read is fresh.
       -> consistent AND available, simultaneously.

09:00-14:00  Still healthy. Millions of requests succeed
       with no trade-off visible at all.

14:03  A network partition splits the cluster in two.
       A write arrives that needs both sides to agree.
       -> ONLY NOW does the system have to choose:
          reject/delay it (favor C) or accept it locally
          on one side only (favor A).

14:03-14:07  Partition persists. The chosen behavior
       (say, CP) applies to affected writes for this
       four-minute window only.

14:07  Partition heals. Cluster reconciles state.
       -> back to consistent AND available together.

3: Before any partition, the system provides both properties at once — this is the normal case for the vast majority of a system's uptime.
10: The trade-off only becomes forced once a partition actually starts and a specific operation needs the unreachable side.
20: The moment the partition heals, the system returns to being both consistent and available — the earlier choice was not permanent.

Why this works: A timeline makes the "only during a partition" scoping concrete — most engineers who say a database "is CP" or "is AP" have correctly summarized its partition-time default, but the misreading is treating that label as true of every second the system runs.

Designing as if a database is missing a property all the time

Wrong

text

"We chose a CP database, so we can never rely
on it being available — we need a fallback for
every single request, all the time."

Better

text

"We chose a database that favors consistency
specifically during a partition. In normal
operation it is both consistent and available;
we only need a fallback/degraded path for the
rare window when a partition is actually
affecting the relevant operation."

What you see: A system is over-engineered with fallback paths, circuit breakers and degraded modes for every request, sized as if partitions were the common case — when in most production clusters a genuine multi-node network partition is a rare event measured in minutes per year, not the default state.

Why: Confusing "the database's partition-time behavior is CP" with "the database is always operating in CP mode" leads teams to pay an ongoing engineering cost for a condition that is actually rare and temporary.

The CAP trade-off is temporary, not permanent

Healthy

start

Partitioned

Healed

end

The misreading vs. what CAP actually constrains

The misreading vs. what CAP actually constrains
Claim	Misreading	What CAP actually says
Scope	Applies to the system at all times	Applies only during an active partition
Which operations	Every operation is affected	Only operations needing cross-partition coordination
Partition tolerance	A third property you can trade away	Not realistically optional — networks do fail
Normal operation	System is permanently missing one property	System is typically both consistent and available

Remember: Partition tolerance is not a free third choice — real networks fail regardless. CAP forces a C-vs-A trade-off only during an actual partition, only for the operations that need the unreachable side; the rest of the time, most systems are both.

CAP in real systems

A worked contrast between a real CP system (etcd/Raft) and a real AP system (Dynamo-style), and the concrete consequence of each.

A real system's CAP choice: leader-based CP vs. multi-writer AP

coreintermediate

Two concrete, well-known systems make opposite defaults for the same partition scenario. etcd (used for Kubernetes cluster state and distributed locks) uses the Raft consensus algorithm and requires a majority (quorum) of nodes to agree before a write is accepted — during a partition, the minority side simply cannot get writes through, so it returns errors instead of stale answers: that is a CP choice. Dynamo-style stores built for availability (the design popularized by Amazon's original Dynamo paper, and reflected in systems like Cassandra) instead accept writes on any reachable replica during a partition and reconcile conflicting versions after the partition heals: that is an AP choice. Neither is "better" in the abstract — they produce different concrete failures for the exact same network event.

Think of it as

It is like two different policies for a shared physical logbook split across two offices that lose contact. The CP office's policy is "no one may write in the book unless a majority of keyholders are present to witness it" — so during the outage, the isolated office simply turns people away rather than risk two conflicting entries. The AP office's policy is "always let people write, note the time, and a librarian will merge the two books later" — so during the outage, both offices keep working, but the librarian may later find two people wrote over the same line and has to decide whose entry wins.

text

CP (etcd/Raft):   write -> needs majority ack -> no quorum -> error
AP (Dynamo-style): write -> accepted locally -> reconciled after heal

What we're doing: Trace the same three-node cluster and the same write attempt through both a CP and an AP system during a partition.

cp-vs-ap-partition.txttext

Cluster: nodes 1, 2, 3 — a partition isolates node 3
         alone from nodes 1 and 2 (which stay together).

--- CP system (e.g. etcd, Raft-based) ---
Client writes to node 3 (the isolated minority):
  - node 3 cannot reach a majority (needs 2 of 3)
  - node 3 REJECTS the write: "no quorum, try again"
  - nodes 1+2 still form a majority and keep accepting
    writes normally
Consequence: clients talking only to node 3 see errors
  until the partition heals or they reach nodes 1/2.
  No client ever reads a stale or conflicting value.

--- AP system (e.g. Cassandra, Dynamo-style) ---
Client writes to node 3 (the isolated minority):
  - node 3 ACCEPTS the write locally, tags it with a
    version/timestamp
  - nodes 1+2 also keep accepting writes to the same key
Consequence: node 3's copy and the 1+2 copies can now
  disagree on the same key. When the partition heals,
  the system runs conflict resolution (e.g. last-write-
  wins or a vector-clock merge) — no client saw an error,
  but the value each client read during the partition
  might not match what "wins" afterward.

8: This is the CP consequence in action: the minority node refuses to commit rather than risk disagreeing with the majority later.
19: This is the AP consequence in action: the same isolated node accepts the write instead, trading a guaranteed-fresh answer for staying available.
23: The reconciliation step is the price of the AP choice — it does not happen for free, and the "losing" write is effectively overwritten after the fact.

Why this works: Running the identical partition through both systems side by side is what makes "different consequence, not different quality" concrete — a team choosing between them is really choosing which failure mode they can tolerate for this specific data.

Picking an AP store for data where a silently overwritten conflicting write is unacceptable

Wrong

text

"We'll use our AP store for account balances —
it's more available, so it's the safer choice."

Better

text

"Account balances need every write to be
seen consistently — a silently-resolved conflict
between two concurrent debits could lose money.
Use a CP store (or a single-writer/quorum design)
for balances; save the AP store for data where a
stale read or a last-write-wins merge is truly
harmless, like a view counter or a cache."

What you see: Two concurrent debits against the same account both succeed on different replicas during a partition, and after reconciliation only one debit survives — the account balance is now wrong in a way no error or log entry flagged at the time it happened.

Why: Availability sounds strictly better than "returns an error," but for data with a correctness invariant (money, inventory, unique identifiers), an AP system's silent post-partition conflict resolution can be worse than a visible, retriable error — the right choice depends on which failure mode the specific data can tolerate.

Same partition, opposite defaults

etcd (CP, Raft)

+Requires majority quorum to commit
+Minority side rejects writes with an error
+No client ever sees conflicting data

Dynamo-style (AP)

−Accepts writes on any reachable replica
−All sides keep working, no errors
−Conflicting versions reconciled after healing

Two real systems, opposite defaults, same partition

Two real systems, opposite defaults, same partition
System	CAP default	Mechanism	Consequence during a partition
etcd (Raft consensus)	CP	Requires majority quorum to commit a write	Minority-side writes fail fast with an error; no conflicting data is ever visible
Dynamo-style store (e.g. Cassandra)	AP	Accepts writes on reachable replicas; reconciles later	All sides keep accepting writes; conflicting versions must be merged once reachable again

Remember: etcd (Raft, needs quorum) rejects writes on the minority side during a partition = CP, fails loud. Dynamo-style stores (Cassandra and similar) accept writes on every reachable side and reconcile later = AP, fails quiet (a conflict to resolve, not an error).