beginnerintermediateadvanced

System Design · Section 22

Replication

Level: intermediate
Read: 18 min
Concepts: 3

A primary/replica architecture keeps one instance responsible for writes while one or more replicas hold a continuously updated copy, replicated either synchronously (safer, slower, no data loss on failover) or asynchronously (faster, can lose the newest writes if the primary fails before replicating). Routing reads to replicas scales read capacity but trades away freshness — most visibly in the read-your-own-writes case — because a replica is only ever as current as its replication lag allows, and that lag is a backlog that grows under write pressure rather than a fixed constant.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

One primary accepts writes; replicas receive and apply a stream of those writes.
Synchronous replication waits for replica acknowledgment before confirming a write; asynchronous confirms immediately.
Read replicas scale read throughput, not write throughput, and every replica read risks returning stale data.
Replication lag is a queue that grows under write bursts and network delay — it needs monitoring, not an assumption of near-zero.

What you will be able to do

Explain the durability trade-off between synchronous and asynchronous replication
Identify the read-your-own-writes problem and a common fix for it
Reason about why replication lag grows and what conditions make it worse

Primary/replica architecture

One writer, many copies, and the durability trade-off in how those copies are kept in sync.

Primary/replica architecture and sync vs async replication

coreintermediate

A primary/replica setup has one instance (the primary) that accepts writes, and one or more replicas that receive a continuous copy of those writes. Synchronous replication waits for a replica to confirm before the primary reports success — safer, slower. Asynchronous replication reports success immediately and streams the change to replicas afterward — faster, with a real chance of losing the most recent writes if the primary fails first.

Think of it as

The primary is the one person allowed to write the master copy of a shared document. Synchronous replication is calling a colleague and waiting for them to confirm they've copied your latest edit before you tell your boss it's done. Asynchronous replication is telling your boss it's done immediately, and separately emailing the colleague the update — usually fine, but if you get hit by a bus between hitting send and the colleague reading it, your latest edit exists nowhere else.

text

write → primary
primary → [sync] wait for replica ack → confirm to client
primary → [async] confirm to client → stream to replica

What we're doing: Show why asynchronous replication can lose the most recent write during a primary failure.

async-replication-failure.txttext

1. Client writes "balance = 500" to the primary.
2. Primary applies the write, confirms "OK" to client.
3. Primary begins streaming the change to the replica.
4. Primary crashes before the replica receives it.
5. The replica is promoted to primary (failover).
6. The replica still shows the old balance —
   the confirmed write from step 2 is gone.

With synchronous replication, step 2's confirmation
would not have happened until the replica already had
the new balance — so this loss could not occur.

2: The client is told the write succeeded here — before any replica has it.
6: This is the actual data loss: a write the client was told succeeded is now unrecoverable.

Why this works: This gap between "confirmed to the client" and "durable on more than one node" is exactly what synchronous replication closes, at the cost of extra write latency — the trade-off has to be made deliberately, not discovered during an incident.

Assuming a replica set automatically means no data loss on failover

Wrong

text

"We have replicas, so failover means we never
lose writes."

Better

text

"We have asynchronous replicas, so a primary
failure can lose the last few unreplicated
writes — acceptable for this workload, but not
a 'zero data loss' guarantee. For data where
that gap isn't acceptable, we'd need synchronous
or semi-synchronous replication instead."

What you see: After a failover, a small number of recently "successful" writes are missing on the newly promoted primary — surprising to a team that assumed replication alone guaranteed durability.

Why: Replication's durability guarantee depends entirely on whether it is synchronous — most systems default to asynchronous replication for performance, which means the guarantee most people assume ("replicated means safe") does not actually hold without an explicit choice to pay the synchronous latency cost.

Primary/replica: sync vs. async confirmation

Client

writes

Primary

accepts the write

Confirm (sync)

after replica ack

Confirm (async)

before replica has it

Replica

receives the stream

Synchronous vs asynchronous replication

Synchronous vs asynchronous replication
Property	Synchronous	Asynchronous
Write latency	Higher — waits for replica ack	Lower — confirms immediately
Data loss on primary failure	None (replica has every acked write)	Possible — unreplicated writes are lost
Availability if a replica is slow/down	Write can stall or fail	Unaffected — replica catches up later
Typical use	Financial/critical data, small clusters	Read replicas at scale, cross-region replicas

Remember: Primary accepts writes, replicas receive a stream of them. Synchronous replication waits for a replica ack before confirming (safer, slower); asynchronous confirms immediately (faster, can lose the newest writes on failover).

Read replicas and the consistency implications

coreintermediate

Read replicas let a system scale reads horizontally by routing SELECT-style queries to replicas instead of the primary, freeing the primary to handle writes. The trade-off is consistency: a replica is only ever as current as its last applied change, so a read routed to a replica can return data that is milliseconds — or, under load, much longer — out of date compared to the primary.

Think of it as

A primary with read replicas is like a company with one person who updates the master price list (the primary) and several people who each keep their own copy to answer customer questions quickly (the replicas). Most of the time their copies match. But if a price just changed, whoever last synced their copy a moment ago is still quoting the old price — not lying, just not yet caught up.

text

writes  → primary
reads   → primary          (need latest data)
        → replica (any)    (staleness acceptable)

What we're doing: Show the classic read-your-own-writes bug caused by routing a post-write read to a lagging replica.

read-your-writes.txttext

1. User updates their profile bio via a write to
   the primary. Primary confirms success.
2. Client immediately re-fetches the profile page,
   and that read is routed to a read replica.
3. The replica hasn't applied the update yet
   (replication lag is 50ms; the read happened
   30ms after the write).
4. The profile page renders the OLD bio — the user
   sees their own just-saved change "disappear."

1: The write itself succeeded — this is not a data-loss bug.
4: The user sees stale data purely because of which node answered the read, not because anything was actually lost.

Why this works: This is the single most common user-facing symptom of read-replica routing done naively — the write is correct and durable, but the very next read can appear to contradict it.

Routing every read to a replica without a read-your-writes exception

Wrong

text

"All reads go to replicas — that's the whole
point of having them."

Better

text

"Reads go to replicas by default, except:
route a user's own read immediately after their
own write to the primary (or a replica confirmed
caught up) for a short window, since that's the
case users actually notice."

What you see: Support tickets or bug reports describing "I saved my changes and they didn't stick" that turn out, on investigation, to be a replica lag issue rather than a real data-loss bug — the write did stick, the read just hit a stale replica.

Why: Not every read needs the primary's freshness, but the specific case of a user reading back their own recent write is the one users notice immediately and report as a bug — worth special-casing even if most other reads are fine on a replica.

Read-your-own-writes, broken by replica lag

Client

Primary

Replica

1. update bio
2. OK
3. GET profile
4. old bioreplica has not applied the update yet

What routing a read to a replica does and does not guarantee

What routing a read to a replica does and does not guarantee
Guarantee	Primary	Read replica
Sees every write immediately	Yes	No — bounded by replication lag
Scales independently of write load	No	Yes — add more replicas
Safe for "read my own recent write"	Yes	Not guaranteed without extra handling
Safe for analytics/reporting reads	Yes, but competes with write load	Yes — isolates that load from the primary

Remember: Read replicas scale reads, not writes, and every replica read is a trade of freshness for capacity — the read-your-own-writes case is the one that most often needs a deliberate exception.

The freshness gap

Why a replica can fall behind, and why that gap is not a fixed, small number.

Why a just-written record may not appear on a lagging replica

standardintermediate

Replication lag is the delay between a write landing on the primary and that same write being applied on a replica. It is caused by network transfer time, the replica applying changes in order, and the replica simply falling behind under heavy write load — it is not a bug, it is an inherent property of asynchronous replication that a design has to account for.

Think of it as

A replica is like a second person copying down what a fast talker (the primary) is saying, word for word. Most of the time they keep up almost instantly. But if the speaker suddenly talks much faster, the note-taker falls further and further behind — not because they stopped listening, but because writing takes real time and the backlog of unwritten words keeps growing until the speaker slows back down.

sql

-- PostgreSQL: check replication lag from the primary
SELECT client_addr, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)
         AS lag_bytes
FROM pg_stat_replication;

What we're doing: Show replication lag growing under a write burst and its effect on read freshness.

replication-lag-growth.txttext

Normal load: primary produces 100 writes/sec,
replica applies 100 writes/sec — lag stays near 0.

Write burst: primary produces 500 writes/sec for
2 minutes, replica can still only apply ~150/sec
(bound by its own disk I/O) — lag grows by roughly
350 writes/sec of backlog for those 2 minutes.

Result: by the time the burst ends, the replica is
tens of seconds behind. Any read routed to it during
or shortly after the burst reflects data from before
the burst even started.

5: This is the actual mechanism: the replica has a hard replay throughput ceiling independent of how fast the primary accepts writes.
9: The user-visible effect — reads reflecting old data — lasts well past the burst itself, until the backlog is fully drained.

Why this works: Lag is not constant — it is a queue that grows under write pressure and drains afterward, so "replicas are usually a few ms behind" can become "replicas are 30 seconds behind" during exactly the traffic spikes when freshness matters most.

Assuming replication lag is always small enough to ignore

Wrong

text

"Replicas are basically real-time — a few
milliseconds of lag doesn't matter for anything
we do."

Better

text

"Replicas are usually a few milliseconds
behind, but lag isn't bounded — it grows under
write bursts, replica resource contention, or
long-running queries on the replica itself.
Monitor lag and have a plan (route to primary,
show a 'data may be delayed' state) for when it
grows past what the workflow can tolerate."

What you see: A dashboard or feature that reads from a replica works fine in normal testing but shows visibly stale data specifically during the traffic spikes or batch jobs that also happen to be generating the most writes — the two are directly connected, not a coincidence.

Why: Lag under normal, low-write conditions and lag under peak write load are different numbers, often by orders of magnitude — a design that only measured the calm-conditions number has not actually characterized its worst case.

Replication lag: normal load vs. a write burst

Normal load

+Primary: 100 writes/sec
+Replica applies 100 writes/sec
+Lag stays near 0

Write burst (2 min)

−Primary: 500 writes/sec
−Replica can only apply ~150/sec (disk I/O bound)
−Lag grows to tens of seconds behind

Remember: Replication lag is a growing-and-draining backlog, not a fixed constant — it worsens under write bursts and replica resource pressure, and needs monitoring, not an assumption that it stays small.