Distributed Time and Clocks

Why real clocks move backward and drift apart, why that breaks timestamp-based ordering, and the four tools that work around it

Clocks driftNTP corrections and drift are normal, not rare faults
Logical clockscausal order with no wall-clock dependency at all
Timeoutsa distributed system cannot tell slow from down

1.Wall-clock time can move; process clocks drift

Wall-clock time is the human-facing notion of "what time is it right now" — and the fact that it "can move" refers to something most application code implicitly assumes cannot happen: a system clock does not only ever advance smoothly forward. NTP (Network Time Protocol) synchronization, which most servers use to keep their clocks accurate, corrects a clock periodically against a reference time source, and that correction can move the clock backward as well as forward if it had drifted ahead — meaning code that reads the wall clock twice in a row and assumes the second reading is always greater than or equal to the first can be wrong. Separately, even between synchronization corrections, every machine's clock runs on its own local oscillator, which is not perfectly accurate — this is clock drift, and it means two machines' clocks, even if synchronized a moment ago, will have diverged by some small amount by the time either is read again, with the exact amount of divergence depending on hardware quality and how recently each was last corrected. Neither of these is a rare hardware fault; both are the normal, expected behavior of real clocks on real machines, in every data center, all the time. Code that treats wall-clock time as instantaneous, monotonically increasing, and identical across machines is making three assumptions that are each false in the general case, and a design that depends on any of them being true needs a different tool for that specific need — which is exactly what the next two concepts in this section provide.

Key features

NTP synchronization corrects a system clock periodically against a reference source, and that correction can move the clock backward, not just forward — code cannot assume two consecutive wall-clock reads are non-decreasing
Clock drift is the natural divergence between two machines' local oscillators, happening continuously even between synchronization corrections — two machines synchronized a moment ago will already have measurably diverged clocks by the next read
Neither NTP correction nor drift is a rare hardware fault — both are the normal, continuous, expected behavior of every real clock on every real machine
The magnitude of drift and correction varies by hardware and network conditions, but in practice can be tens of milliseconds to low seconds between synchronizations on typical commodity hardware — enough to matter for anything comparing timestamps at fine granularity across machines
A single machine's wall clock going backward can break code that assumes elapsed time (end_time - start_time) is always non-negative, or that a later-created record always has a numerically larger creation timestamp
This is a distinct problem from the previous section's "arrival order is not production order" — clock drift means even the timestamps themselves, set at true production time on different machines, may not accurately reflect the true relative order of events

Two distinct clock behaviors and what each one breaks

Two distinct clock behaviors and what each one breaks
Behavior	What happens	What it can break in code
NTP correction ("wall-clock time can move")	Clock is nudged forward or backward to match a reference source	Code assuming time.now() is non-decreasing across two reads
Clock drift ("process clocks can drift")	Two machines' clocks slowly diverge between corrections	Code comparing timestamps from different machines as if from one clock

A wall clock over time: drift, then an NTP correction moves it backward

T+0
Clock synchronized
matches reference time exactly
T+30min
Drift accumulates
local oscillator runs slightly fast; clock is now ahead
T+60min
NTP correction
clock is stepped backward to match reference time
T+60min+ε
A reading taken just after
can be earlier than a reading taken just before the correction

Command

python

# An assumption that silently breaks when the wall
# clock moves backward mid-measurement
start = time.time()
do_work()
end = time.time()
duration = end - start  # can be NEGATIVE if an NTP
                          # correction moved the clock
                          # backward between the two reads

A wall clock can move backward (NTP corrections) and two machines' wall clocks continuously drift apart between corrections (clock drift) — both are the normal, expected behavior of real clocks, not rare faults. Never use the wall clock to measure elapsed duration (use a monotonic clock instead) and never assume two machines' wall-clock timestamps are precise enough for an exact ordering decision, even when both run NTP in the same data center.

2.Do not rely on local timestamps for global ordering

Given that wall-clock time can move and drift between machines (the prior concept), the direct consequence for system design is that comparing two events' local timestamps — each set by whichever machine produced that specific event — cannot reliably tell you their true relative order when the two events happened close together in time, and "close together" in practice can mean tens or even hundreds of milliseconds, not just microseconds. This matters specifically when the two events come from different machines; a single machine's own sequence of local timestamps is generally reliable relative to itself (barring an NTP correction landing in the exact wrong instant), but as soon as two different machines' clocks are involved, their independent drift and independent NTP correction schedules mean neither can vouch for the other's exact instant. This is exactly the failure mode illustrated in the version-check concept in the prior section: two updates to the same order, produced by two different services with clocks that disagree by even a few milliseconds, can have their true order reversed by comparing timestamps, silently applying a stale update over a newer one. The fix is not "get better clocks" — synchronization can reduce the disagreement but never eliminate it to zero, and the disagreement only has to exceed the gap between two real events to cause an ordering error. The fix is to use a mechanism that does not depend on clock agreement at all: a sequence number, a version check, or (for the general distributed-systems case) a logical clock, covered in the next concept.

Key features

A single machine's own consecutive local timestamps are generally reliable relative to each other; the unreliable case is comparing timestamps across two different machines
Two events produced close together in time (within the actual clock disagreement between the two machines, commonly tens of milliseconds or more) can have their true order flipped by comparing local timestamps
Better clock synchronization reduces the size of the disagreement but cannot eliminate it — the failure mode only requires the disagreement to exceed the real gap between the two events being compared, which can always happen for events close enough together
This is the same failure this project's own version-check concept demonstrated directly: a shipping service's clock running a few seconds fast made its "shipped" event appear, by timestamp, to happen after a "cancelled" event that actually happened later
The reliable alternatives do not depend on clock agreement at all: a sequence number (a monotonic counter) or a version check (compare and reject anything not strictly newer) both sidestep the problem entirely, as covered in the prior section
Even Google Spanner's TrueTime — an unusually sophisticated, hardware-assisted clock synchronization system — does not claim to eliminate clock uncertainty; it bounds it explicitly and has the database wait out the bound before committing, rather than pretending the uncertainty is zero

When comparing local timestamps across machines is and is not reliable

When comparing local timestamps across machines is and is not reliable
Situation	Reliable?	Why
Comparing two timestamps from the same machine	Generally yes	One clock, one source of drift, mostly self-consistent
Comparing two timestamps from different machines, seconds apart	Usually yes	Typical clock disagreement (ms) is far smaller than the gap
Comparing two timestamps from different machines, milliseconds apart	No	Clock disagreement can be the same order of magnitude as the gap itself

Ordering by local timestamp vs. by a clock-independent signal

Compare local timestamps

Reliable only when events are far apart relative to clock disagreement
Cross-machine comparisons inherit both machines' drift
Better synchronization narrows but never eliminates the risk
Fails silently — a flipped order looks like a valid timestamp comparison

Compare a sequence number / version

No dependency on any clock at all
Correct regardless of how close together the events happened
Requires the producer to actually maintain the counter
This project's standard recommendation wherever ordering matters

Command

python

# Unreliable: ordering two updates from different
# services by their local timestamps
if event_a.timestamp < event_b.timestamp:
    apply_first(event_a)
else:
    apply_first(event_b)
# if event_a and event_b's producing machines'
# clocks disagree by more than the true gap between
# the two events, this can pick the wrong order

# Reliable: ordering by a version/sequence that
# does not depend on any clock at all
if event_a.version < event_b.version:
    apply_first(event_a)
else:
    apply_first(event_b)

Comparing local timestamps across different machines is unreliable specifically when the events being compared happen close together relative to the machines' actual clock disagreement — and better synchronization narrows that risk without eliminating it. Use a clock-independent signal (a sequence number, a version check, or a logical clock for genuine cross-machine causal ordering) wherever an ordering decision needs to be correct by construction rather than correct most of the time. Never trust a client-supplied timestamp for a server-side ordering decision.

3.UTC, monotonic clocks and logical ordering

Four distinct tools close out this section, each solving a different piece of the time-and-ordering problem the prior two concepts raised. UTC (Coordinated Universal Time) is the standard, timezone-free reference for wall-clock time — storing and exchanging every timestamp in UTC rather than a local timezone eliminates an entire, separate class of bugs (daylight saving transitions, ambiguous local times, servers in different timezones disagreeing about "what time is it") that has nothing to do with clock drift but is just as damaging if ignored. A monotonic clock is a clock guaranteed to only ever move forward, never backward and never jump due to an NTP correction — every mainstream language exposes one specifically for measuring elapsed duration (Python's time.monotonic(), Java's System.nanoTime()), and it should be the default choice any time code computes "how much time has passed" rather than "what time is it." Logical ordering (logical clocks — Lamport timestamps and vector clocks are the two classic mechanisms) captures causal relationships between events across machines without depending on wall-clock agreement at all: a Lamport timestamp increments on every event and is updated to be greater than any timestamp an event carries when received, which guarantees that if event A causally influenced event B, A's logical timestamp is smaller than B's — a property no wall clock can guarantee across machines. And timeouts, understood conceptually here (with the mechanics covered in full in Timeouts and Resource Limits), are the practical acknowledgment that a distributed system can never distinguish "the other side is slow" from "the other side is down" with certainty, so every cross-machine wait needs an explicit bound rather than an assumption that a response will eventually arrive.

Key features

UTC has no timezone offset and no daylight saving transitions — storing and transmitting timestamps in UTC (converting to a local timezone only for display) eliminates an entire class of bugs unrelated to clock drift
A monotonic clock only ever moves forward and is unaffected by NTP corrections — it should be the default for measuring elapsed duration; the wall clock should be reserved for "what time is it right now" questions
A Lamport timestamp is a single integer counter that captures a partial causal order: if event A happened-before event B (A causally influenced B), A's Lamport timestamp is guaranteed smaller than B's — but the reverse is not guaranteed, so two Lamport timestamps being ordered does not prove one caused the other
A vector clock extends this to detect concurrent (causally unrelated) events precisely, at the cost of a counter per participating node rather than a single integer — used where "these two events are causally unrelated" needs to be distinguishable from "one caused the other," not just where global ordering is needed
A timeout exists because a distributed system fundamentally cannot distinguish "slow" from "down" without an explicit bound — no amount of waiting resolves that ambiguity, it only delays the decision to treat the situation as a failure
These four tools solve four different problems and are frequently used together on the same system: UTC for stored/human-facing timestamps, a monotonic clock for measuring durations, a logical clock for causal ordering across machines, and timeouts for bounding every cross-machine wait

Four tools and the specific problem each one solves

Four tools and the specific problem each one solves
Tool	Problem it solves	What it does not solve
UTC	Timezone ambiguity and daylight-saving bugs	Clock drift between machines
Monotonic clock	Measuring elapsed duration without backward jumps	What time it actually is (no wall-clock meaning)
Logical clock (Lamport/vector)	Capturing causal order across machines without relying on wall clocks	Human-readable "what time did this happen"
Timeout	Bounding an otherwise-indefinite wait for a cross-machine response	Distinguishing "slow" from "down" with certainty — it only forces a decision

Four tools, four distinct jobs

UTC

one universal reference, no timezone ambiguity

Monotonic clock

measures duration, never steps backward

Logical clock

causal order, no wall-clock dependency

Timeout

bounds an otherwise-indefinite wait

Command

python

# All four tools used together for the job each
# is actually good at
created_at_utc = datetime.now(timezone.utc)   # UTC:
                                                # stored,
                                                # human-facing
start = time.monotonic()                       # monotonic:
do_work()                                      # elapsed
elapsed = time.monotonic() - start             # duration

lamport_clock = max(lamport_clock, received_ts) + 1
# logical clock: causal ordering, no wall-clock
# dependency

response = call_with_timeout(dependency, seconds=3)
# timeout: bound the wait; treat "no answer by 3s"
# as a failure rather than waiting indefinitely

UTC removes timezone and daylight-saving ambiguity from stored timestamps; a monotonic clock is the correct tool for measuring elapsed duration, never the wall clock; a logical clock (Lamport timestamps, vector clocks) captures causal order across machines without depending on wall-clock agreement at all; and a timeout is the explicit acknowledgment that a distributed system cannot tell "slow" from "down," so every cross-machine wait needs a bound. These four tools solve four different problems and are commonly used together on the same system, not as alternatives to each other.

Spotted a mistake on this page? Tell us.

Read Sheet