Filter concepts by levelShowing all levels.

System Design · Section 78

Rate of Change and Schema Evolution

Level
intermediate
Read
16 min
Concepts
4

A running system is never fully at rest — APIs, event schemas, database schemas and application code all change while old versions of each are still in active use, and this section is the discipline of making that coexistence safe rather than assuming every consumer, instance, or reader upgrades in lockstep. API and event versioning distinguishes additive, backward-compatible changes (which need no version bump, since any existing consumer can ignore a new optional field) from breaking changes (which need an explicit new version served alongside the old one during a deprecation window, so every consumer — including ones a producer may not have full visibility into — migrates on its own schedule); the discipline is identical for a request/response API and for events published to a queue or log, though events often have less visible consumers. Database schema changes need the expand-contract pattern specifically because a rolling deployment means old and new application code query the same live schema simultaneously for the length of the rollout: expand by adding new structure alongside old, migrate by deploying code that works with both while backfilling data, and contract by removing the old structure only once verified evidence confirms nothing reads it anymore — a one-step rename or drop breaks whichever code version does not match it. Rolling deployments generalize this: any change has to work correctly across all four combinations of old and new instances interacting — old-to-old, new-to-new, old-to-new, new-to-old — because that mixed state persists for the entire rollout window, not a brief instant. And event schema evolution formalizes the safety bar with two independent properties, backward compatibility (a new schema can read data written under an old one) and forward compatibility (an old schema can read data written under a new one) — a genuinely safe change like an additive field with a default preserves both at once, and a full-history replay job, which exposes every schema version a log has ever held in one pass, is what makes backward compatibility a strict requirement rather than a nice-to-have.

System Design overview

What is true here

  1. Additive changes need no version bump; breaking changes need a new version plus a deprecation window where both versions are served.
  2. The identical versioning discipline applies to published events, not just request/response APIs — often with less visibility into who the consumers actually are.
  3. Expand-contract (add new structure → dual-write and migrate → remove old structure only after verified zero-use) is what makes a breaking database change safe during a rolling deployment.
  4. A rolling deployment means old and new instances coexist and interact for the entire rollout — a change must work across all four old/new combinations, not just the fully-migrated end state.
  5. Event schemas need both backward compatibility (new schema reads old data) and forward compatibility (old schema reads new data) — a full-history replay exposes every schema version at once, making backward compatibility strict.

What you will be able to do

  • Decide whether a given API or event change needs a version bump, and design an appropriate deprecation window for one that does
  • Apply the expand-contract pattern to a breaking database schema change so it survives a rolling deployment
  • Identify which of the four old/new instance interaction combinations a given change has not yet been verified against
  • Distinguish backward from forward compatibility for an event schema change, and explain why a replay job requires the stricter transitive compatibility check

Versioning the contract

Deciding when a change needs a version bump, and giving consumers a real deprecation window for the ones that do.

Versioning APIs and events carefully

coreintermediate

An API or an event schema is a contract between whoever produces it and every consumer that reads it, and unlike a function inside one codebase, that contract cannot be changed and redeployed everywhere at once — a mobile app version from six months ago, a partner's integration, and an internal service that has not been redeployed yet may all still be calling the old contract long after a new one ships. Versioning is the discipline of managing that reality deliberately rather than assuming every consumer upgrades in lockstep with the producer. The safest changes are additive and backward-compatible — adding a new optional field, adding a new endpoint — which every existing consumer can simply ignore without being broken. A breaking change (removing a field, changing a field's type or meaning, changing required parameters) needs an explicit versioning strategy: a new version number in the URL or a header, coexisting for a defined deprecation window with the old version, so consumers migrate on their own schedule rather than breaking the instant the change ships. The same discipline applies to events published onto a queue or log (Kafka, SQS): a consumer reading an event schema has the same lockstep problem as an API caller, except often with less visibility into who all the consumers even are, since a queue or topic can have consumers a producer does not know about at all.

Think of it as

Think of a public library revising the layout of its card catalog. If it only adds new categories (additive, backward-compatible), a patron's decades-old habit of looking up fiction under "F" still works exactly as before, plus new categories exist for people who want them. If the library removes the "F" section entirely and reorganizes everything under a new scheme overnight (a breaking change with no versioning), every patron who walks in the next day using their old habit is lost — the library has broken a contract patrons had no way to know was about to change and no time to adapt to. A well-versioned change is the library posting "the fiction section is moving to the new wing on the first of next month, both layouts work until then" — giving every patron, on their own visit schedule, a window to adjust.

http
# Two APIs live at once during a deprecation window
GET /v1/orders/42
{"id": 42, "total": 59.99}

GET /v2/orders/42
{"id": 42, "total": {"amount": 5999, "currency": "USD"}}
# v1 kept serving unchanged until its documented
# sunset date; v2 is where the breaking change
# (total's type changed from a plain number to an
# object) actually lives

What we're doing: Compare shipping a breaking field-type change with no versioning versus with a proper deprecation window.

breaking-change-comparison.txttext
Change: "total" field changes from a plain number
(59.99) to an object ({amount: 5999, currency: "USD"})

No versioning:
  - Change deployed directly to /v1/orders
  - Every existing consumer parsing total as a number
    breaks the instant this deploys, with no warning

Versioned, with deprecation window:
  - New shape shipped as /v2/orders
  - /v1/orders keeps returning the old shape unchanged
  - Deprecation notice + sunset date published
  - Consumers migrate to /v2 over a 90-day window
  - /v1 retired only after the window elapses
6
This is the failure mode versioning exists to prevent: every consumer that assumed the old shape breaks simultaneously, at a time chosen entirely by the producer, with no ability for any individual consumer to control when it is affected.
13
Every consumer migrates on its own schedule within the window — a mobile app that only gets a new release every few weeks and an internal service that redeploys daily both have a real path to migrate without either one being broken in the meantime.

Why this works: The underlying change (total's type) is identical in both cases — the difference is entirely in whether consumers were given a contract that let them adapt, and that difference is the entire practical value of a deliberate versioning strategy over an ad hoc "just ship the improved shape" approach.

Treating a field-type change as safe because "most clients probably don't care"

Wrong

text
# Reasoning used to skip versioning a breaking
# change: "the new shape is objectively better,
# and most of our clients are internal services
# we can just ask to redeploy quickly."

Better

text
# Version the change regardless of how
# confident the team is about "most" consumers --
# a mobile app store review cycle, a partner's
# integration, or an internal service simply not
# yet redeployed are all real consumers a
# breaking change can silently take down, and
# "most" is never "all."

What you see: A breaking change shipped without versioning "because most consumers would be fine" breaks a partner integration that had not been touched in months, and the partner discovers it only when their own customers start reporting errors — well after the change already shipped.

Why: "Most clients probably don't care" is a judgment about the clients a team knows about and is thinking of at the moment of shipping — it says nothing about clients that are less visible (an old partner integration, a cached mobile app version still in the wild, an internal service nobody remembered still calls this endpoint), which are exactly the ones a versioning strategy protects without requiring the producer to have complete visibility into every consumer.

A breaking change, versioned and given a deprecation window
  1. T+0

    v2 released

    v1 continues unchanged; both live side by side

  2. T+0 to T+90d

    Deprecation window

    consumers migrate to v2 on their own schedule

  3. T+60d

    Sunset warning

    v1 responses include a deprecation header

  4. T+90d

    v1 retired

    only after the documented window elapses

  1. T+0: v2 released — v1 continues unchanged; both live side by side
  2. T+0 to T+90d: Deprecation window — consumers migrate to v2 on their own schedule
  3. T+60d: Sunset warning — v1 responses include a deprecation header
  4. T+90d: v1 retired — only after the documented window elapses

Change type, and whether it needs a version bump

Change type, and whether it needs a version bump
ChangeBackward-compatible?Needs a new version?
Add a new optional fieldYesNo
Add a new endpoint / event typeYesNo
Remove a fieldNoYes — old version stays live during a deprecation window
Change a field's type or meaningNoYes
Make an optional field requiredNoYes
Change an endpoint's or event's success/error semanticsNoYes

Remember: Additive, backward-compatible changes need no version bump — existing consumers can ignore what they do not recognize. A breaking change needs an explicit new version, served alongside the old one for a defined deprecation window, so every consumer — including the ones a producer does not have full visibility into — migrates on its own schedule rather than breaking the instant the change ships. This applies to events on a queue or log exactly as much as to a request/response API.

See also: backward compatible database changes · event schema evolution and consumer compatibility · versioning and backward compatibility

Advertisement

Evolving shared state and running code together

Expand-contract for database changes, and designing for old and new instances to coexist for an entire rollout.

Backward-compatible database changes

coreintermediate

A database schema is read by application code that is not deployed all at once — during any rolling deployment, old application code and new application code query the same database simultaneously for at least a short window, which means a schema change has to work correctly against both versions of the code at once, not just the new one. This is a harder version of the API-versioning problem, because a database usually cannot serve "two versions of the schema" side by side the way an API can serve /v1 and /v2 — there is one live schema, and every currently-running instance of the application, old or new, is reading and writing against it at the same instant. The standard technique for making a genuinely breaking change (renaming a column, changing its type, splitting one column into several) safe is the expand-contract pattern: expand the schema by adding the new structure alongside the old one, deploy application code that writes to both and reads from whichever is authoritative, wait for every old instance to be replaced by new instances that only need the new structure, then contract by removing the old structure once nothing reads it anymore. Skipping straight to the "final" schema — dropping a column or renaming it in one migration — works fine in a single-instance, single-deploy world and breaks immediately in a rolling deployment, because for the length of the rollout, some fraction of running instances are still querying a column that no longer exists.

Think of it as

Renovating one lane of traffic on a bridge that must stay open in both directions the whole time: you cannot simply demolish the old lane and build the new one in its place, because traffic (old and new application instances) has to keep flowing across the bridge the entire time the work is happening. The actual technique is building the new lane alongside the old one first (expand), routing some traffic onto it while the old lane is still open (both schemas live simultaneously), waiting until every vehicle that specifically needed the old lane has crossed (every old instance has been replaced), and only then closing and removing the old lane (contract). Demolishing the old lane on day one, before the new one is ready, strands every car still relying on it — exactly the outcome a one-step schema migration produces for old application instances still running during a rollout.

sql
-- Expand-contract for renaming "full_name" to
-- "display_name" across a rolling deployment

-- Phase 1: Expand — add the new column, backfill it
ALTER TABLE users ADD COLUMN display_name TEXT;
UPDATE users SET display_name = full_name;

-- Phase 2: Migrate — new app instances write to both
-- columns; old instances keep working against full_name
-- unchanged. Wait until every instance is on the new code.

-- Phase 3: Contract — only once no instance reads
-- full_name anymore
ALTER TABLE users DROP COLUMN full_name;

What we're doing: Compare a one-step column rename against the expand-contract version during a rolling deployment of 10 instances.

rename-during-rollout.txttext
Rolling deployment: 10 instances, replaced 1 at a
time, ~2 minutes apart (20 minutes total rollout)

One-step migration (rename full_name -> display_name
in a single ALTER, deployed alongside new app code):
  t=0    migration runs, column renamed instantly
  t=0    9 of 10 instances are still running OLD code
         querying "full_name" -- every one of them
         starts erroring immediately

Expand-contract:
  t=0    display_name added (expand); old code
         unaffected, still reads/writes full_name
  t=0-20m  instances replaced one at a time with new
         code that dual-writes both columns
  t=20m  all 10 instances confirmed on new code;
         zero reads against full_name for 24h
  t=44h  full_name dropped (contract) -- safe, because
         nothing has needed it for a verified window
8
This is the exact failure the expand-contract pattern exists to prevent: the migration and the code deploy are not atomic together across a fleet, so any window where old code and a changed schema coexist is a window where old code breaks — and that window is the entire rollout, not an instant.
17
The "48h" style wait for the contract phase, based on verified evidence of zero old-column access, is what actually makes dropping the old column safe — it isn't a fixed calendar rule but a confirmation step.

Why this works: The database change is conceptually the same (rename a column) in both timelines, but only the expand-contract version accounts for the actual mechanics of how a rolling deployment works — application code and schema changes are never atomic together across a whole fleet, and any migration strategy that assumes they are will break during every single rollout, not just unlucky ones.

Dropping a column in the same migration that stops using it

Wrong

sql
-- Single migration, deployed alongside new code
-- that no longer references the old column
ALTER TABLE orders RENAME COLUMN qty TO quantity;
-- old application instances still running during
-- the rollout query "qty" and error immediately

Better

sql
-- Expand: add the new column first, as its own
-- migration, deployed well before any app code
-- change
ALTER TABLE orders ADD COLUMN quantity INTEGER;
UPDATE orders SET quantity = qty;
-- new app code dual-writes; old app code is
-- entirely unaffected because "qty" still exists
-- and is still updated

What you see: Every request handled by an instance that has not yet redeployed starts throwing a database error the instant the migration runs, because the column it queries no longer exists — during a 20-minute rolling deployment, this means roughly 90% of traffic is affected for the first two minutes and a shrinking fraction after that, all avoidable.

Why: A migration and an application deployment are two separate events that a rolling-deployment strategy deliberately does not synchronize precisely — assuming they land atomically together across every instance treats a distributed, staggered rollout as if it were a single, instantaneous cutover, which it structurally is not.

Expand-contract: renaming a column safely during a rolling deploy

Expand

add display_name alongside full_name; nothing removed yet

Dual-write

new code writes both columns; old code still works unchanged

Backfill

copy existing full_name values into display_name

Verify

confirm zero reads/writes against full_name from any instance

Contract

drop full_name only after verification

  1. Expand — add display_name alongside full_name; nothing removed yet
  2. Dual-write — new code writes both columns; old code still works unchanged
  3. Backfill — copy existing full_name values into display_name
  4. Verify — confirm zero reads/writes against full_name from any instance
  5. Contract — drop full_name only after verification

Expand-contract: three phases for a database change that would otherwise break old code

Expand-contract: three phases for a database change that would otherwise break old code
PhaseWhat happensSafe to remove old structure?
ExpandAdd the new column/table alongside the old one; nothing removedNo — old code still needs the old structure
MigrateDeploy code that writes to both, reads from whichever is authoritative; backfill existing dataNo — rollout to all instances is still in progress
ContractRemove the old structureYes — only after confirming zero reads/writes against it

Remember: A rolling deployment means old and new application code query the same live schema simultaneously for the length of the rollout — a one-step rename or drop breaks whichever code version does not match it. The expand-contract pattern (add the new structure, dual-write and migrate data while both old and new code run, remove the old structure only after verified confirmation nothing uses it anymore) is what makes a genuinely breaking database change safe during a real, staggered rollout.

See also: rolling deployments and coexisting versions · api and event versioning

Rolling deployments: designing for coexisting versions

coreintermediate

A rolling deployment replaces instances of a service one at a time (or in small batches) rather than stopping every instance and starting the new version simultaneously, specifically to avoid downtime — but the direct consequence is that for the entire duration of the rollout, some fraction of instances are running the old code and some are running the new code, at the same time, serving the same traffic and talking to the same shared dependencies. A design has to actively support that coexistence rather than assume it away: two instances of the same service, one old and one new, might both pull a message off the same shared queue, both write to the same database, or both be behind the same load balancer receiving requests round-robin — and the system needs to behave correctly regardless of which combination of versions ends up interacting with each other at any given moment. This means every change has to be evaluated not just as "does the new version work" but as "does the new version work correctly while some requests are still being served by the old version, reading and writing the same shared state." A change that is perfectly fine once every instance is on the new version, but broken during the mixed period, is not actually safe to ship as a rolling deployment — it needs the same versioning/expand-contract discipline applied to APIs and databases elsewhere in this section.

Think of it as

A relay race where the runners are being swapped out mid-race, one at a time, without ever stopping the race itself: at any given moment there is a mix of "old model" and "new model" runners on the track together, all still passing the baton back and forth to whichever runner happens to be next, regardless of which model they are. If the new runners use a subtly different baton-passing technique that only works runner-to-runner among themselves, the race falls apart the moment an old-model runner has to pass to (or receive from) a new-model one — which will happen constantly during the swap, not just at its very start or end. Supporting rolling deployment is designing the baton pass so it works for every combination — old-to-old, new-to-new, old-to-new, and new-to-old — because during the actual swap, all four combinations happen.

text
# A rolling deployment of 10 instances, and the
# actual state of the world at the midpoint:
t=10min (rollout 50% complete):
  instances 1-5: new code (v2)
  instances 6-10: old code (v1)
  load balancer: routes new requests to any of the
    10, regardless of version, round-robin
  shared queue: messages produced by v1 AND v2
    instances are interleaved in the same queue,
    consumed by whichever instance (v1 or v2) is
    next available
  shared database: read and written by both
    versions simultaneously, the entire time

What we're doing: Trace a message produced by an old instance and consumed by a new instance mid-rollout, and see what a format assumption breaks.

consumer-assumes-new-format.pypython
# v1 (old instance) still running mid-rollout,
# publishes messages in the old format:
def publish_order_event(order):
    queue.publish({"order_id": order.id, "total": order.total})

# v2 (new instance) consumer code, deployed assuming
# EVERY message is already in the new format:
def handle_order_event(message):
    amount = message["total"]["amount"]  # KeyError:
    # "total" is still a plain number in messages
    # published by any v1 instance still running
4
This v1 publisher is still running and still producing messages in the old format for the entire duration of the rollout — it has no way to know a new consumer format exists, and it should not need to.
9
The new consumer assumes every message it reads was produced by an already-migrated publisher, which is false for as long as any v1 instance remains — exactly the coexistence window a rolling deployment guarantees will exist.

Why this works: The new consumer code is correct for the eventual, fully-migrated steady state and incorrect for the actual state of the world during the rollout — which is the state the code runs in for the entire deployment window, not a rare edge case, so this is a near-certain failure on every rollout rather than an occasional one.

Deploying a consumer that only handles the new message format

Wrong

python
def handle_order_event(message):
    amount = message["total"]["amount"]
    currency = message["total"]["currency"]
    process(amount, currency)

Better

python
def handle_order_event(message):
    total = message["total"]
    if isinstance(total, dict):        # new format
        amount, currency = total["amount"], total["currency"]
    else:                               # old format
        amount, currency = total, "USD"  # old messages
        # were always USD; document that assumption
    process(amount, currency)

What you see: Order-processing errors spike to roughly the same percentage as the fraction of old instances still running, for the entire duration of every rollout, because the new consumer code can only handle messages from already-migrated producers and a meaningful fraction of producers have not migrated yet at any given moment during the rollout.

Why: A consumer deployed as part of the same rolling rollout as the producers it reads from cannot assume the producers it reads from have already finished rolling out too — deployments of different services (or even different instances of the same service) are not synchronized events, and code that assumes they are breaks for the entire, often lengthy, period where that assumption is false.

Old and new instances coexisting mid-rollout
Load Balancer
Old Instance (v1)
New Instance (v2)
Shared Queue
Shared DB
  1. 1. route request A
  2. 2. route request B
  3. 3. publish message (v1 format)
  4. 4. consume messagemust handle v1-format messages too
  5. 5. write using new schema
  6. 6. read same rowmust still work against the new schema
  1. Load Balancer → Old Instance (v1): route request A
  2. Load Balancer → New Instance (v2): route request B
  3. Old Instance (v1) → Shared Queue: publish message (v1 format)
  4. New Instance (v2) → Shared Queue: consume message (must handle v1-format messages too)
  5. New Instance (v2) → Shared DB: write using new schema
  6. Old Instance (v1) → Shared DB: read same row (must still work against the new schema)

Four combinations a rolling deployment must handle correctly

Four combinations a rolling deployment must handle correctly
CombinationWhen it happensWhat can go wrong if unhandled
Old talking to oldBefore the rollout starts, and for any instance not yet replacedNothing new — this is the pre-rollout steady state
New talking to newAfter the rollout completesNothing new — this is the post-rollout steady state
Old talking to newAn old instance reads data/messages written by a new instanceOld code cannot parse a new field/format it does not expect
New talking to oldA new instance reads data/messages written by an old instanceNew code expects a field/format the old instance never wrote

Remember: A rolling deployment means old and new instances coexist and both actively serve traffic and share state for the entire rollout window, not just for an instant — a change is only safe to ship this way if it works correctly across all four combinations (old-to-old, new-to-new, old-to-new, new-to-old), which is exactly what the versioning and expand-contract disciplines in this section exist to guarantee. Never assume a client's consecutive calls, or a consumer's messages, come from an already-fully-migrated fleet.

See also: backward compatible database changes · event schema evolution and consumer compatibility · load balancing algorithms · autoscaling triggers and metrics

Advertisement

Event schema compatibility

Backward and forward compatibility as two independent properties, and why replay makes backward compatibility strict.

Event schema evolution and consumer compatibility

coreintermediate

Events published to a log or queue (Kafka, SQS, an event bus) often outlive the exact moment they were produced — a consumer might process an event seconds after it was published, or a replay might reprocess events from months ago, and a schema registry or topic frequently accumulates events written under several different schema versions over its lifetime, all still sitting there to be read. This makes event schema evolution stricter than API versioning in one specific way: an API client makes a new request every time and can be pointed at a new version, but an event consumer often has to be able to read every version of the schema that could still exist in the stream, including old ones written long before the consumer's current code was deployed. Two specific compatibility properties formalize what "safe evolution" means. Backward compatibility means a new schema can read data written with an old schema — a consumer upgraded to understand the new schema can still process old events. Forward compatibility means an old schema can read data written with a new schema — a consumer that has not yet upgraded can still process new events, typically by ignoring fields it does not recognize. A genuinely safe change (adding an optional field with a default, for instance) is compatible in both directions at once; a change that breaks either direction (removing a field a consumer still expects, changing a field's type) needs the same deprecation-window treatment as an API, except now measured against the consumer with the oldest deployed code and the oldest events still eligible to be replayed, not just the current moment.

Think of it as

A shared filing cabinet that has been in continuous use for ten years, where documents were filed under a slightly different form template every few years, and multiple people with different training (using different versions of the instructions) all pull folders from it on any given day. Backward compatibility is a newly-trained clerk who can still correctly read a folder filed a decade ago under the old form. Forward compatibility is an old-school clerk, trained only on the original form, who can still correctly extract what they need from a folder filed yesterday using this year's updated form (perhaps because the updated form only added a new box at the bottom, which the old-school clerk simply never looks at and does not need to). A filing system redesigned so that neither kind of clerk can read the other era's folders means someone, at some point, has to reprocess the entire decade of paperwork by hand — which is exactly the operational cost a genuinely evolvable event schema is designed to avoid.

json
// Event written under schema v1 (still sitting in
// the log, replayable)
{"order_id": 42, "total": 59.99}

// Event written under schema v2 (currency added,
// optional, with an implied default)
{"order_id": 43, "total": 64.99, "currency": "USD"}

// A backward-compatible v2 consumer reading the v1
// event above supplies the default itself:
// currency = event.get("currency", "USD")

// A forward-compatible v1 consumer reading the v2
// event above simply ignores the unrecognized
// "currency" field and proceeds normally

What we're doing: Trace an event-replay job reading a log spanning three schema versions, and see what happens without backward compatibility.

replay-across-schema-versions.txttext
Event log spans 3 schema versions over 2 years:
  v1 events (year 1): {"order_id": N, "total": N}
  v2 events (year 2, added optional "currency"):
      {"order_id": N, "total": N, "currency": "USD"}
  v3 events (this year, "total" changed to an
      object): {"order_id": N, "total": {"amount":
      N, "currency": "USD"}}

Replay job (rebuilding an analytics store from the
full 2-year history) runs the current (v3-aware)
consumer against the entire log:
  v3 events: read correctly (native format)
  v2 events: read correctly if the consumer treats
    "total" as either a number or an object
  v1 events: read correctly only if the consumer
    ALSO supplies a default currency for events
    that never had the field at all
13
This is backward compatibility in action across two schema generations at once: the current consumer has to correctly interpret both v2's plain-number total and v1's complete absence of a currency field, not just the single most recent prior version.
15
A consumer written to only handle "the current schema, plus one prior version" would fail here — a full historical replay exposes every schema version the log has ever contained, not just the two most recent ones.

Why this works: A replay job is the scenario where event schema evolution is tested most severely, because it reads the entire historical range of schema versions in one pass rather than the narrow window of versions active at any single moment — a design that only considered "the current consumer talking to the current producer" would look correct for months and then fail the first time anyone actually tries to replay from the beginning.

Changing a field's type in place, assuming consumers will "just handle it"

Wrong

json
// v1 events already in the log:
{"order_id": 42, "total": 59.99}
// v2 change: same field name, new type, deployed
// with no schema-registry compatibility check
{"order_id": 43, "total": {"amount": 6499, "currency": "USD"}}

Better

json
// Add a NEW field instead of changing the type of
// an existing one, and deprecate the old field
// over a window, exactly like an API:
{"order_id": 43, "total": 64.99,
 "total_v2": {"amount": 6499, "currency": "USD"}}
// consumers migrate to reading total_v2 on their
// own schedule; total is dropped only once nothing
// reads it

What you see: Every consumer still expecting `total` to be a number — including any replay job processing the full historical log, which is mostly v1-shaped events — throws a type error the moment it encounters a v2 event, and there is no way to distinguish "old event, safe to assume a number" from "new event, must be treated as an object" without inspecting the schema version explicitly, which the field itself does not carry.

Why: Changing a field's type in place is one of the changes that breaks compatibility in both directions at once — old consumers cannot parse the new shape (not forward-compatible) and new consumers cannot assume a uniform shape when reading old events without extra type-checking logic that a same-name field gives no natural way to trigger (not cleanly backward-compatible either), which is exactly why introducing a new field name for the new shape, rather than repurposing the old one, is the safer path.

Plotting a schema change by which compatibility it preserves
Add optional field w/ default
safe in both directions
Remove a field
old consumers still expecting it break
Make a field required
old consumers fail validation on new events
Change a field's type
breaks both directions
  • Add optional field w/ default: Preserves forward compatibility, Preserves backward compatibility — safe in both directions
  • Remove a field: Preserves forward compatibility, Breaks backward compatibility — old consumers still expecting it break
  • Make a field required: Breaks forward compatibility, Preserves backward compatibility — old consumers fail validation on new events
  • Change a field's type: Breaks forward compatibility, Breaks backward compatibility — breaks both directions

Backward vs forward compatibility, and what breaks each

Backward vs forward compatibility, and what breaks each
CompatibilityWhat it guaranteesCommon change that breaks it
BackwardNew schema can read old dataRemoving a field new code assumed was optional to skip, but actually still needs a value for
ForwardOld schema can read new dataMaking a new field required, so old code fails validation on an event it does not understand
Both (full compatibility)Any consumer version can read any event versionOnly additive changes with defaults, and never removing/renaming a field

Remember: Backward compatibility (new schema reads old data) and forward compatibility (old schema reads new data) are two independent properties, and a genuinely safe change — an additive field with a default — preserves both at once. Event schemas need this discipline more strictly than APIs because a log or topic can hold events spanning years of schema versions, and a replay job exposes the full historical range at once, not just the versions active right now — use transitive compatibility checking, not just pairwise, for any topic that supports full-history replay.

See also: api and event versioning · replayable event logs · tolerating duplicates and out of order events

Advertisement