Rate of Change and Schema Evolution

Key features

• Additive, backward-compatible changes (new optional fields, new endpoints) do not need a version bump — every existing consumer can ignore what it does not recognize
• A breaking change (removing/renaming a field, changing its type or required-ness, changing an endpoint's semantics) needs an explicit version strategy — a new API version, or a new event schema version — with the old one still served during a deprecation window
• Common API versioning strategies are a version segment in the URL (/v2/orders), a version header, or content negotiation — the choice matters less than picking one and applying it consistently
• Events have the same lockstep problem as APIs but often worse visibility: a topic or queue can have consumers the producer does not know about, so "just tell everyone about the breaking change" is not always possible the way it might be for a documented, registered set of API clients
• A deprecation window (a defined period where both the old and new version are served) is what actually makes a breaking change safe — a breaking change shipped with no old version served at all forces every consumer to migrate at the exact moment the producer chooses, not the moment each consumer is ready
• Semantic versioning conventions (major.minor.patch) map cleanly onto this: a major version bump signals a breaking change, a minor bump signals a backward-compatible addition — communicating the nature of a change, not just that a change happened

Command

# Two APIs live at once during a deprecation window
GET /v1/orders/42
{"id": 42, "total": 59.99}

GET /v2/orders/42
{"id": 42, "total": {"amount": 5999, "currency": "USD"}}
# v1 kept serving unchanged until its documented
# sunset date; v2 is where the breaking change
# (total's type changed from a plain number to an
# object) actually lives

Key features

• During a rolling deployment, old and new application code query the same live database schema simultaneously — there is no equivalent of an API's /v1 and /v2 living side by side at the database layer by default
• The expand-contract pattern has three phases: expand (add new structure alongside old, without removing anything), migrate (deploy application code that works with both, backfill data into the new structure), contract (remove the old structure only once nothing reads it)
• A one-step migration that renames or drops a column works only if every application instance querying the database is updated at the exact same instant — true for very few real deployments, and false for essentially all rolling/blue-green deployments
• Adding a nullable column, or a column with a default, is backward-compatible on its own — old code that does not know about the new column simply never references it and continues to work unchanged
• Renaming a column is not backward-compatible on its own even though it looks like a small change — old code querying the old name breaks the instant the rename lands, so it needs the expand-contract treatment: add the new name, dual-write during migration, remove the old name only once nothing reads it
• The contract phase should only run once monitoring confirms zero reads/writes against the old structure — removing it based on an assumption ("the rollout should be done by now") rather than verified evidence risks breaking an instance that has not actually redeployed yet

Command

-- Expand-contract for renaming "full_name" to
-- "display_name" across a rolling deployment

-- Phase 1: Expand — add the new column, backfill it
ALTER TABLE users ADD COLUMN display_name TEXT;
UPDATE users SET display_name = full_name;

-- Phase 2: Migrate — new app instances write to both
-- columns; old instances keep working against full_name
-- unchanged. Wait until every instance is on the new code.

-- Phase 3: Contract — only once no instance reads
-- full_name anymore
ALTER TABLE users DROP COLUMN full_name;

Key features

• During a rolling deployment, old and new instances coexist and both actively serve traffic, read/write shared state, and interact with shared queues and databases — this is not a brief instant, it typically spans the entire rollout window
• A change is only safe for a rolling deployment if it works correctly in all combinations: old instance talking to old, new to new, old to new, and new to old — not just in the fully-new steady state
• A message queue or event topic is a common place this bites: an old consumer and a new consumer can both be pulling from the same queue simultaneously, so a message format change needs the same backward-compatible treatment as an API or database change
• Load balancers typically route to instances round-robin or by least-connections, with no awareness of which application version an instance is running — a client cannot assume which version handled its previous request will handle its next one
• Feature flags are a common complement to rolling deployments: a feature can be deployed as dormant code across all instances first, then activated via a flag once every instance is confirmed on the new version, decoupling "code is deployed" from "feature is active" so behavior does not depend on the exact rollout progress
• The same discipline extends to background workers and scheduled jobs, not just request-serving instances — a worker fleet mid-rollout has exactly the same old/new coexistence problem as an API fleet

Command

# A rolling deployment of 10 instances, and the
# actual state of the world at the midpoint:
t=10min (rollout 50% complete):
  instances 1-5: new code (v2)
  instances 6-10: old code (v1)
  load balancer: routes new requests to any of the
    10, regardless of version, round-robin
  shared queue: messages produced by v1 AND v2
    instances are interleaved in the same queue,
    consumed by whichever instance (v1 or v2) is
    next available
  shared database: read and written by both
    versions simultaneously, the entire time

Key features

• Backward compatibility: new schema can read old data — a consumer running the newest code can still correctly process events written under an older schema still sitting in the log or replayable from storage
• Forward compatibility: old schema can read new data — a consumer still running older code can still correctly process events written under a newer schema, typically by ignoring unrecognized new fields
• A field addition with a sensible default is compatible in both directions: old consumers ignore the new field (forward-compatible), and new consumers supply the default when reading old events that lack it (backward-compatible)
• Removing a field or changing its type breaks compatibility in at least one direction and needs the deprecation-window treatment — but the relevant window is bounded by both the oldest still-deployed consumer AND the oldest event still eligible for replay, which can be a much longer window than a typical API deprecation period
• A schema registry (used with Avro, Protobuf, or JSON Schema) enforces compatibility rules automatically at publish time — rejecting a producer's schema change that would violate the configured compatibility mode, catching an unsafe change before it ever reaches consumers
• Event replay (reprocessing an entire historical log, e.g. to rebuild a derived data store) makes backward compatibility a hard requirement, not a nice-to-have — a consumer that cannot read the oldest schema version in the log cannot correctly replay from the beginning

Command

// Event written under schema v1 (still sitting in
// the log, replayable)
{"order_id": 42, "total": 59.99}

// Event written under schema v2 (currency added,
// optional, with an implied default)
{"order_id": 43, "total": 64.99, "currency": "USD"}

// A backward-compatible v2 consumer reading the v1
// event above supplies the default itself:
// currency = event.get("currency", "USD")

// A forward-compatible v1 consumer reading the v2
// event above simply ignores the unrecognized
// "currency" field and proceeds normally