beginnerintermediateadvanced

System Design · Section 80

Distributed Time and Clocks

Level: intermediate
Read: 14 min
Concepts: 3

Real clocks behave in two ways application code routinely assumes away: wall-clock time can move backward when an NTP correction steps a drifted clock back toward a reference source, and every machine's clock drifts continuously between corrections due to its own imperfect local oscillator — both are the normal, expected operation of real clocks on real machines, not rare hardware faults. The direct consequence is that comparing local timestamps from different machines cannot reliably establish true event order when the events happened close together relative to the actual clock disagreement, which can be tens of milliseconds or more even between well-synchronized servers in the same data center — and better synchronization narrows that risk without ever eliminating it, since the failure only requires the disagreement to exceed the real gap between two specific events. Four tools close out the practical response to this: UTC removes an entirely separate class of ambiguity (timezones, daylight-saving transitions) from stored and exchanged timestamps; a monotonic clock, guaranteed to only ever move forward, is the correct tool for measuring elapsed duration, a job the wall clock cannot safely do; a logical clock (Lamport timestamps or vector clocks) captures genuine causal order across machines using only integers exchanged alongside real messages, with no dependency on any clock's accuracy at all; and a timeout is the practical acknowledgment that a distributed system can never distinguish "the other side is slow" from "the other side is down" with certainty, so every cross-machine wait needs an explicit bound rather than an assumption that a response will eventually arrive.

This section

Master Vector Clocks: Logical Timestamps for Distributed Systems & System Design InterviewsArpit Vaish

What is true here

A wall clock can step backward (NTP corrections) and drift continuously relative to other machines' clocks — both are normal, expected behavior, not rare faults.
Comparing local timestamps across machines is unreliable for events close together relative to the actual clock disagreement, and better synchronization only narrows, never eliminates, that risk.
A monotonic clock, not the wall clock, is the correct tool for measuring elapsed duration — only a monotonic clock is guaranteed never to step backward.
A logical clock (Lamport timestamps, vector clocks) captures causal order across machines without depending on any clock's accuracy at all.
A timeout exists because a distributed system cannot distinguish "slow" from "down" — every cross-machine wait needs an explicit bound, since the absence of one is a silent choice to wait indefinitely.

What you will be able to do

Explain why wall-clock time moving backward and clock drift are normal operating conditions, not rare faults
Identify when comparing local timestamps across machines is and is not reliable for an ordering decision
Choose a monotonic clock over the wall clock for measuring elapsed duration
Explain how a Lamport timestamp captures causal order without relying on wall-clock synchronization

How real clocks actually behave

Wall-clock corrections and drift as normal operation, and the direct consequence for cross-machine timestamp comparison.

Wall-clock time can move; process clocks drift

coreintermediate

Wall-clock time is the human-facing notion of "what time is it right now" — and the fact that it "can move" refers to something most application code implicitly assumes cannot happen: a system clock does not only ever advance smoothly forward. NTP (Network Time Protocol) synchronization, which most servers use to keep their clocks accurate, corrects a clock periodically against a reference time source, and that correction can move the clock backward as well as forward if it had drifted ahead — meaning code that reads the wall clock twice in a row and assumes the second reading is always greater than or equal to the first can be wrong. Separately, even between synchronization corrections, every machine's clock runs on its own local oscillator, which is not perfectly accurate — this is clock drift, and it means two machines' clocks, even if synchronized a moment ago, will have diverged by some small amount by the time either is read again, with the exact amount of divergence depending on hardware quality and how recently each was last corrected. Neither of these is a rare hardware fault; both are the normal, expected behavior of real clocks on real machines, in every data center, all the time. Code that treats wall-clock time as instantaneous, monotonically increasing, and identical across machines is making three assumptions that are each false in the general case, and a design that depends on any of them being true needs a different tool for that specific need — which is exactly what the next two concepts in this section provide.

Think of it as

A room full of wall clocks, each slightly cheap and slightly different, all periodically walked past and nudged closer to the "true" time by someone carrying a reference clock — sometimes that nudge sets a clock forward, and sometimes, if it had been running fast, the nudge sets it backward. Between nudges, every clock in the room keeps ticking at its own very slightly wrong rate, so even two clocks nudged to agree exactly a minute ago will not read exactly the same time now. Nobody would build a stopwatch out of "the reading on whichever wall clock happens to be nearest" — not because wall clocks are broken, but because that was never the job a wall clock does well; a stopwatch needs a mechanism that only ever counts forward from a fixed start, which is a fundamentally different kind of device.

python

# An assumption that silently breaks when the wall
# clock moves backward mid-measurement
start = time.time()
do_work()
end = time.time()
duration = end - start  # can be NEGATIVE if an NTP
                          # correction moved the clock
                          # backward between the two reads

What we're doing: Trace a duration measurement across an NTP correction and see it go negative.

negative-duration-trace.txttext

10:00:00.000  start = wall_clock_read()   # 10:00:00.000
              (work begins; clock has drifted
              slightly ahead of true time)
10:00:00.050  NTP daemon steps the clock backward
              by 80ms to correct accumulated drift
10:00:00.020  end = wall_clock_read()    # 10:00:00.020,
              which is EARLIER than start, even
              though real elapsed time was positive
              duration = end - start = -0.030s

4: This step is the normal, periodic behavior of an NTP client keeping the machine's clock accurate — it is not a malfunction, and it happens on essentially every server running standard time synchronization.
8: The measured duration is negative despite real, physical time having genuinely passed between the two reads — the wall clock, not the passage of time itself, moved backward, and code trusting the wall clock as a stopwatch inherits that discontinuity.

Why this works: The bug here is not in the NTP correction, which is doing exactly its job (keeping the clock accurate) — the bug is in using `time.time()` (a wall clock, designed to tell you what time it is) for a job it was never designed for (measuring elapsed duration, which needs a clock that only ever counts forward).

Measuring elapsed duration with the wall clock

Wrong

python

start = time.time()      # wall clock
do_work()
elapsed = time.time() - start   # can be negative

Better

python

start = time.monotonic()  # monotonic clock,
do_work()                  # never steps backward
elapsed = time.monotonic() - start  # always >= 0

What you see: A rate limiter or a timeout mechanism that computes "has more than 5 seconds passed" using the wall clock occasionally reports a negative or absurdly large elapsed time right after an NTP correction, causing a request to be incorrectly allowed through a limiter that should have blocked it, or a timeout to fire (or fail to fire) at the wrong moment.

Why: The wall clock and the monotonic clock answer two genuinely different questions — "what time is it" versus "how much time has passed" — and only the second question has an answer that must never go backward; using the wall clock (built for the first question) to answer the second inherits every discontinuity the wall clock is allowed to have.

A wall clock over time: drift, then an NTP correction moves it backward

T+0
Clock synchronized
matches reference time exactly
T+30min
Drift accumulates
local oscillator runs slightly fast; clock is now ahead
T+60min
NTP correction
clock is stepped backward to match reference time
T+60min+ε
A reading taken just after
can be earlier than a reading taken just before the correction

Two distinct clock behaviors and what each one breaks

Two distinct clock behaviors and what each one breaks
Behavior	What happens	What it can break in code
NTP correction ("wall-clock time can move")	Clock is nudged forward or backward to match a reference source	Code assuming time.now() is non-decreasing across two reads
Clock drift ("process clocks can drift")	Two machines' clocks slowly diverge between corrections	Code comparing timestamps from different machines as if from one clock

Remember: A wall clock can move backward (NTP corrections) and two machines' wall clocks continuously drift apart between corrections (clock drift) — both are the normal, expected behavior of real clocks, not rare faults. Never use the wall clock to measure elapsed duration (use a monotonic clock instead) and never assume two machines' wall-clock timestamps are precise enough for an exact ordering decision, even when both run NTP in the same data center.

Do not rely on local timestamps for global ordering

coreintermediate

Given that wall-clock time can move and drift between machines (the prior concept), the direct consequence for system design is that comparing two events' local timestamps — each set by whichever machine produced that specific event — cannot reliably tell you their true relative order when the two events happened close together in time, and "close together" in practice can mean tens or even hundreds of milliseconds, not just microseconds. This matters specifically when the two events come from different machines; a single machine's own sequence of local timestamps is generally reliable relative to itself (barring an NTP correction landing in the exact wrong instant), but as soon as two different machines' clocks are involved, their independent drift and independent NTP correction schedules mean neither can vouch for the other's exact instant. This is exactly the failure mode illustrated in the version-check concept in the prior section: two updates to the same order, produced by two different services with clocks that disagree by even a few milliseconds, can have their true order reversed by comparing timestamps, silently applying a stale update over a newer one. The fix is not "get better clocks" — synchronization can reduce the disagreement but never eliminate it to zero, and the disagreement only has to exceed the gap between two real events to cause an ordering error. The fix is to use a mechanism that does not depend on clock agreement at all: a sequence number, a version check, or (for the general distributed-systems case) a logical clock, covered in the next concept.

Think of it as

Two witnesses in different rooms of a large building, each wearing a slightly different, unsynchronized wristwatch, asked afterward "which of these two things happened first — the loud bang in your room, or mine?" If the two events happened five minutes apart, both witnesses' watches will agree on the order even if they disagree on the exact time by a minute or two. But if the two events happened one second apart, whichever witness's watch happens to be running fast or slow at that moment can easily make the two reported times come out in the wrong order — not because either witness lied, but because a wristwatch was never precise enough, relative to another independent wristwatch, to resolve an ordering question at that fine a timescale. The fix is not asking the witnesses to buy more expensive watches; it is asking a third question that does not depend on either watch at all — "did room A's microphone hear room B's bang before or after room A's own bang," which is a fact about the events themselves, not about anyone's clock.

python

# Unreliable: ordering two updates from different
# services by their local timestamps
if event_a.timestamp < event_b.timestamp:
    apply_first(event_a)
else:
    apply_first(event_b)
# if event_a and event_b's producing machines'
# clocks disagree by more than the true gap between
# the two events, this can pick the wrong order

# Reliable: ordering by a version/sequence that
# does not depend on any clock at all
if event_a.version < event_b.version:
    apply_first(event_a)
else:
    apply_first(event_b)

What we're doing: Compare two update events one millisecond apart, ordered by timestamp versus by version, when the producing machines' clocks disagree by 5ms.

timestamp-vs-version-ordering.txttext

Machine A's clock: running 5ms ahead of true time
Machine B's clock: accurate

True order of events (what actually happened):
  1. Machine B produces UpdateX at true time T
  2. Machine A produces UpdateY at true time T+1ms

Recorded local timestamps:
  UpdateX: timestamp = T          (machine B, accurate)
  UpdateY: timestamp = T+1ms+5ms = T+6ms (machine A,
           5ms fast)

Ordering by timestamp: UpdateX (T) before UpdateY
(T+6ms) -- happens to agree with true order here

Recorded versions (each producer's per-entity
sequence, no clock involved):
  UpdateX: version = 4
  UpdateY: version = 5
Ordering by version: UpdateX (4) before UpdateY (5)
-- correct by construction, regardless of any clock

11: The timestamp-based ordering happens to be correct here, but only because machine A's 5ms skew was in the direction that did not flip the outcome — a 5ms skew in the OTHER direction, or a slightly larger skew in this same direction relative to a smaller true gap between the events, would have reversed it.
19: The version-based ordering is correct by construction and does not depend on which direction or how large any clock's skew happens to be — it never needed to reason about clocks at all.

Why this works: This is the general shape of every timestamp-ordering bug: the failure is not guaranteed on every comparison, only on the ones where the clock disagreement happens to exceed the true gap between events and happens to point in the direction that flips the outcome — which makes it a genuinely intermittent, hard-to-reproduce bug rather than a consistent one, and exactly the kind of bug a clock-independent signal eliminates by construction rather than by luck.

Merging events from multiple regions by local timestamp to reconstruct a global order

Wrong

python

all_events = region_a_events + region_b_events + region_c_events
all_events.sort(key=lambda e: e.local_timestamp)
# treats the merged, sorted list as the true
# global order events actually happened in

Better

python

# Where a genuine cross-region causal order is
# needed, use a logical clock (e.g. a Lamport
# timestamp or vector clock) that captures actual
# causal relationships, not wall-clock proximity
all_events.sort(key=lambda e: e.logical_clock)
# or: accept that cross-region events with no
# causal relationship have no meaningful "true"
# order, and stop trying to reconstruct one

What you see: A cross-region audit report periodically shows an event from region C apparently happening before an event in region A that region C's own event was causally triggered by (a reply to a message that, by the report's own timestamp ordering, had not been sent yet) — an impossible ordering that undermines trust in the whole report.

Why: Sorting by local timestamp across regions assumes every region's clock reports the same instant identically, which is precisely the assumption clock drift and independent NTP correction schedules make false — a genuinely causal cross-region ordering question needs a mechanism (a logical clock) that tracks actual causality, not one that hopes wall clocks happen to agree closely enough.

Ordering by local timestamp vs. by a clock-independent signal

Compare local timestamps

+Reliable only when events are far apart relative to clock disagreement
+Cross-machine comparisons inherit both machines' drift
+Better synchronization narrows but never eliminates the risk
+Fails silently — a flipped order looks like a valid timestamp comparison

Compare a sequence number / version

−No dependency on any clock at all
−Correct regardless of how close together the events happened
−Requires the producer to actually maintain the counter
−This project's standard recommendation wherever ordering matters

When comparing local timestamps across machines is and is not reliable

When comparing local timestamps across machines is and is not reliable
Situation	Reliable?	Why
Comparing two timestamps from the same machine	Generally yes	One clock, one source of drift, mostly self-consistent
Comparing two timestamps from different machines, seconds apart	Usually yes	Typical clock disagreement (ms) is far smaller than the gap
Comparing two timestamps from different machines, milliseconds apart	No	Clock disagreement can be the same order of magnitude as the gap itself

Remember: Comparing local timestamps across different machines is unreliable specifically when the events being compared happen close together relative to the machines' actual clock disagreement — and better synchronization narrows that risk without eliminating it. Use a clock-independent signal (a sequence number, a version check, or a logical clock for genuine cross-machine causal ordering) wherever an ordering decision needs to be correct by construction rather than correct most of the time. Never trust a client-supplied timestamp for a server-side ordering decision.

The four tools

UTC, monotonic clocks, logical clocks and timeouts — each solving a different piece of the time-and-ordering problem.

UTC, monotonic clocks and logical ordering

coreintermediate

Four distinct tools close out this section, each solving a different piece of the time-and-ordering problem the prior two concepts raised. UTC (Coordinated Universal Time) is the standard, timezone-free reference for wall-clock time — storing and exchanging every timestamp in UTC rather than a local timezone eliminates an entire, separate class of bugs (daylight saving transitions, ambiguous local times, servers in different timezones disagreeing about "what time is it") that has nothing to do with clock drift but is just as damaging if ignored. A monotonic clock is a clock guaranteed to only ever move forward, never backward and never jump due to an NTP correction — every mainstream language exposes one specifically for measuring elapsed duration (Python's time.monotonic(), Java's System.nanoTime()), and it should be the default choice any time code computes "how much time has passed" rather than "what time is it." Logical ordering (logical clocks — Lamport timestamps and vector clocks are the two classic mechanisms) captures causal relationships between events across machines without depending on wall-clock agreement at all: a Lamport timestamp increments on every event and is updated to be greater than any timestamp an event carries when received, which guarantees that if event A causally influenced event B, A's logical timestamp is smaller than B's — a property no wall clock can guarantee across machines. And timeouts, understood conceptually here (with the mechanics covered in full in Timeouts and Resource Limits), are the practical acknowledgment that a distributed system can never distinguish "the other side is slow" from "the other side is down" with certainty, so every cross-machine wait needs an explicit bound rather than an assumption that a response will eventually arrive.

Think of it as

Four different tools in a toolbox, each built for a job the others cannot do. UTC is a single, universal ruler everyone agrees to measure against, so "3pm" means the same physical instant to everyone reading it, regardless of what timezone they happen to be standing in — as opposed to each city using its own ruler with its own zero point, which is how timezone bugs happen. A monotonic clock is a stopwatch: it only counts forward from when you started it, and pressing "lap" twice can never give you a smaller reading than before, unlike a wall clock someone might reset while you were timing something. A logical clock is not a clock in the ordinary sense at all — it is a shared numbering convention where "I only give out a number higher than any number I have seen so far," which guarantees that a reply always gets a higher number than the message it replied to, regardless of what any actual clock anywhere says. And a timeout is simply drawing a line and saying "if I have not heard back by here, I will treat this as failed and move on" — because waiting forever is itself a choice, and usually the wrong one, when there is no way to know whether "forever" will ever actually end.

python

# All four tools used together for the job each
# is actually good at
created_at_utc = datetime.now(timezone.utc)   # UTC:
                                                # stored,
                                                # human-facing
start = time.monotonic()                       # monotonic:
do_work()                                      # elapsed
elapsed = time.monotonic() - start             # duration

lamport_clock = max(lamport_clock, received_ts) + 1
# logical clock: causal ordering, no wall-clock
# dependency

response = call_with_timeout(dependency, seconds=3)
# timeout: bound the wait; treat "no answer by 3s"
# as a failure rather than waiting indefinitely

What we're doing: Trace a Lamport timestamp across three causally related events on two machines.

lamport-timestamp-trace.txttext

Machine A, local Lamport counter starts at 0
Machine B, local Lamport counter starts at 0

1. Machine A does local work: counter -> 1
   (event A1, lamport_ts=1)
2. Machine A sends a message to B, attaching its
   current counter (1)
3. Machine B receives the message: sets its own
   counter to max(its current 0, received 1) + 1 = 2
   (event B1, lamport_ts=2)
4. Machine B does more local work: counter -> 3
   (event B2, lamport_ts=3)

9: This is the rule that makes the guarantee hold: on receiving a message, a node sets its counter to one more than the larger of its own current value and the value the message carried — which forces the receiving event's timestamp to be strictly greater than the sending event's, regardless of what either machine's wall clock says.
11: B1 (timestamp 2) is guaranteed to have a smaller Lamport timestamp than B2 (timestamp 3), and both are guaranteed greater than A1 (timestamp 1) — capturing that A1 causally preceded B1 (via the message) and B1 preceded B2 (via local sequencing), using nothing but integers exchanged alongside real messages, no clock synchronization involved.

Why this works: The Lamport timestamps correctly capture "A1 happened-before B1" (real causality, since B1 only happened because of the message A1's send triggered) using a mechanism that never once consulted either machine's wall clock — this is the entire value proposition of a logical clock: causal ordering that clock drift, NTP corrections, and cross-machine clock disagreement cannot touch, because none of those things are inputs to it at all.

Storing timestamps in local server time instead of UTC

Wrong

python

# Server running in US Eastern time, storing
# local timestamps
created_at = datetime.now()  # naive, local time,
# ambiguous during a daylight-saving "fall back"
# transition (the same local time occurs twice)

Better

python

created_at = datetime.now(timezone.utc)
# unambiguous, no daylight-saving transitions,
# convert to a display timezone only when
# rendering for a human

What you see: Two records created an hour apart, both timestamped "1:30 AM" in local time because the fall-back daylight-saving transition happened in between, sort adjacent or even in the wrong order when a report naively orders by the stored local timestamp, because the same clock-face time genuinely occurred twice that night.

Why: Local time is not a strictly increasing sequence even on a single, perfectly accurate clock — the twice-yearly daylight-saving transition makes some local times ambiguous (occurring twice) or nonexistent (skipped), a problem entirely independent of clock drift or NTP, and storing in UTC sidesteps it because UTC has no daylight-saving transitions at all.

Four tools, four distinct jobs

UTC

one universal reference, no timezone ambiguity

Monotonic clock

measures duration, never steps backward

Logical clock

causal order, no wall-clock dependency

Timeout

bounds an otherwise-indefinite wait

Four tools and the specific problem each one solves

Four tools and the specific problem each one solves
Tool	Problem it solves	What it does not solve
UTC	Timezone ambiguity and daylight-saving bugs	Clock drift between machines
Monotonic clock	Measuring elapsed duration without backward jumps	What time it actually is (no wall-clock meaning)
Logical clock (Lamport/vector)	Capturing causal order across machines without relying on wall clocks	Human-readable "what time did this happen"
Timeout	Bounding an otherwise-indefinite wait for a cross-machine response	Distinguishing "slow" from "down" with certainty — it only forces a decision

Remember: UTC removes timezone and daylight-saving ambiguity from stored timestamps; a monotonic clock is the correct tool for measuring elapsed duration, never the wall clock; a logical clock (Lamport timestamps, vector clocks) captures causal order across machines without depending on wall-clock agreement at all; and a timeout is the explicit acknowledgment that a distributed system cannot tell "slow" from "down," so every cross-machine wait needs a bound. These four tools solve four different problems and are commonly used together on the same system, not as alternatives to each other.