beginnerintermediateadvanced

System Design · Section 102

Trade-Off Analysis

Level: intermediate
Read: 18 min
Concepts: 2

Every significant decision gets four sentences: the benefit (what improves, with a number where one exists), the cost (what gets worse — money, latency, staleness, operational work, a rule you can no longer enforce in one place), the alternative you did not take, and the reason this trade was right for this system at this scale. Most write-ups carry only the benefit, which is why they read as advocacy. The cost is what makes the decision honest — a decision with no stated cost has either not been examined or is not a trade-off — and stating it as an abstract noun like "added complexity" does not count, because nothing in it could turn out to be wrong. The alternative is what makes the decision revisitable eighteen months later, when someone would otherwise switch to it for reasons that were already known and rejected. Six axes come up in almost every design and are worth having rehearsed: PostgreSQL versus DynamoDB, shared Redis versus a local cache, synchronous versus asynchronous, monolith versus microservices, strong versus eventual consistency, and fan-out on write versus on read. None of them has a winner — each has a cheaper-to-operate default and a condition that justifies leaving it, and the condition is almost always a number: a write rate, a maximum fan-out, a latency target, a tolerable staleness window. Knowing the six cold is what lets a design conversation skip straight to the part that is specific to the system in front of you.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Four parts per decision: benefit, cost, alternative, reason — the cost is what separates analysis from advocacy.
A cost stated as an abstract noun cannot be weighed; name the network call, the saga, the extra on-call surface.
The rejected alternative is what stops the decision being re-litigated or silently reversed later.
The six recurring axes are dials, not switches; each has a default side and an exit condition.
The exit condition is a number — write rate, maximum fan-out, tolerable staleness — not a preference.

What you will be able to do

Write any design decision in the benefit / cost / alternative / reason form, with a number in the benefit and a mechanism in the cost
Recognise a cost line that cannot be reviewed, and rewrite it into one a reviewer could reject
Place a workload on each of the six recurring axes and name the measurement that put it there
Explain why copying a well-known system's answer without its numbers reproduces the answer and not the reasoning

Stating a decision

The four-part form, and the two parts nearly every write-up leaves out.

Benefit, cost, alternative, reason

coreintermediate

Every significant design decision gets four sentences. The benefit: what improves, in a number where one exists. The cost: what gets worse, also concretely — money, latency, staleness, operational work, a rule you can no longer enforce in one place. The alternative: the option you did not take. The reason: why this benefit was worth this cost here, for this system, at this scale. Most write-ups carry only the benefit, which is why they read as advocacy rather than analysis. The cost is what makes the decision honest, and stating it is not a weakness in the proposal — a decision with no stated cost has either not been examined or is not a trade-off at all. The alternative is what makes the decision revisitable: eighteen months later, someone asking "why is it built this way" can see what was already considered and rejected, instead of re-running the whole argument from scratch or, worse, switching to the alternative for reasons that were already known and dismissed. And the reason has to be specific to this system. "Microservices scale better" is a fact about microservices. "We split billing out because it needs a different deploy cadence and a different on-call rotation than the rest" is a reason.

Think of it as

A receipt, not a sales pitch. A receipt names what you got, what you paid, and it exists so that later — when someone asks whether the purchase made sense — the answer does not depend on anyone remembering. The alternative is the item you put back on the shelf, written down so nobody carries it to the till again next month.

text

Decision: serve profile reads from a replica

BENEFIT      p99 profile read 240ms -> 30ms;
             ~70% of reads leave the primary
COST         reads can lag the primary by up
             to 8s during bulk imports
ALTERNATIVE  keep every read on the primary
             and scale it vertically instead
REASON       profiles tolerate a stale view;
             checkout does not, so checkout
             keeps reading the primary

Four sentences. Reviewable by someone who
was not in the room.

What we're doing: Turn a one-line proposal into a decision record a reviewer can disagree with.

decision-record.txttext

BEFORE
  "Let's move notification sending to a
   queue. It'll make the API faster."

That is a benefit and nothing else. Nobody
can agree or disagree with it, because
nothing was put at risk.

AFTER

Decision: move notification sending out of
the request path and onto a work queue.

BENEFIT
  POST /orders no longer waits on the email
  provider. Measured: provider p99 is 1.9s,
  so the endpoint's p99 drops from ~2.3s to
  ~0.4s. The endpoint also stops failing
  when the provider is down.

COST
  Sending becomes eventually consistent. A
  user who checks their inbox immediately
  may see nothing. Queue depth, worker
  health and a dead-letter queue all become
  things we now have to monitor and operate.
  Debugging a missing email means reading a
  queue, not a request trace.

ALTERNATIVE
  Keep the call inline and wrap it in a
  short timeout with one retry, accepting
  that some orders send no email at all.

REASON
  Order creation is the transaction that
  must not fail; the email is a courtesy
  that may arrive seconds late. The
  alternative keeps the coupling and still
  loses emails, so it pays part of the cost
  for none of the benefit.

REVISIT WHEN
  Notifications become part of a legal or
  contractual guarantee, at which point
  "seconds late" stops being acceptable and
  the delivery path needs its own SLO.

15: The benefit carries two numbers and their source — a measured provider p99 and the resulting endpoint p99. A reviewer can challenge either one, which is the point.
20: The cost includes operational work, not just a technical property. New things to monitor and a harder debugging path are real costs that proposals routinely leave out.
33: The reason explains why the alternative loses on its own terms rather than dismissing it. That is what stops the team from re-proposing it in six months.
40: A "revisit when" line is optional but cheap: it converts a decision that will silently expire into one with a stated trigger.

Why this works: The rewritten version is not longer for the sake of ceremony. Each part removes a specific future failure: the benefit gives something to measure against, the cost stops the eventual-consistency surprise from reading as a bug, the alternative stops it being re-litigated, and the reason ties the choice to this system rather than to a general belief about queues.

Writing the reasoning after the decision has shipped

Wrong

text

# Ship first, document in the quarter's
# architecture review:
#   "We chose Kafka because it gives us
#    replay and decoupling."
# True today. But nobody can now recall
# whether replay was a requirement or a
# feature discovered afterwards.

Better

text

# Write the four parts before building,
# in the pull request or the design doc:
#   BENEFIT/COST/ALTERNATIVE/REASON
# Then the record says which requirement
# drove it, and the next decision can be
# checked against the same requirement.

What you see: Every past decision has a rationale, every rationale sounds sensible, and none of them predict anything — because each was written to explain what already exists rather than to choose between options that were still open.

Why: Reconstructed reasoning is a guess about your own past. It reliably drops the alternatives, because the one that was chosen is the only one still visible, and it reliably converts discovered benefits into stated goals. The record then teaches the next decision the wrong lesson.

The four parts of a stated trade-off

BENEFIT p99 profile read 240ms -> 30ms COST reads may lag the primary by 8s ALTERNATIVE scale the primary vertically REASON profiles tolerate stale; checkout does not

BENEFIT

What improves — Named with a number wherever one exists. "Much faster" cannot be checked later; "240 ms to 30 ms" can.

COST

What gets worse — The part most write-ups omit. Without it the document is advocacy, and the cost still arrives — just as a production surprise instead of a decision.

ALTERNATIVE

What you did not take — Makes the decision revisitable. A future reader sees what was already considered instead of switching to it for reasons already known.

REASON

Why the trade was right here — Specific to this system and this scale. A general property of the technology is a fact, not a reason.

The four parts, and what their absence looks like months later

The four parts, and what their absence looks like months later
Part	What it must name	What its absence looks like
Benefit	The property that improves, with a number where one exists	A change nobody can evaluate, because nothing said what it was for
Cost	What gets worse: money, latency, staleness, ops work, a rule split across services	A surprise in production that reads as a bug and was actually a choice
Alternative	The option considered and not taken	The team switches to it later for reasons that were already known and rejected
Reason	Why this benefit beat this cost for this system at this scale	A rule of thumb applied to a system it does not fit

The same decision, stated two ways

The same decision, stated two ways
Part	Advocacy	Analysis
Benefit	Read replicas make the app faster	Moves ~70% of reads off the primary; p99 profile read 240 ms → 30 ms
Cost	(not stated)	Replica lag is normally under 1 s and has been seen at 8 s during bulk imports
Alternative	(not stated)	Keep all reads on the primary and add a bigger instance instead
Reason	Replicas are best practice	Profile reads tolerate an 8 s stale view; checkout does not, so checkout stays on the primary

Remember: Four sentences per significant decision: benefit (what improves, with a number where one exists), cost (what gets worse — money, latency, staleness, operational work), alternative (what you did not take), reason (why this trade was right for this system at this scale). The cost is what makes it analysis rather than advocacy; the alternative is what makes it revisitable. Write it at decision time — reasoning reconstructed later drops the alternatives and turns discovered benefits into stated goals.

The recurring axes

Six dials that appear in almost every design, each with a default and the number that justifies moving it.

Six trade-offs worth having rehearsed

coreintermediate

Six decisions come up in almost every design, and each one is an axis rather than a contest with a winner. PostgreSQL versus DynamoDB trades flexible queries and joins against predictable scale-out on a known key. Shared Redis versus a local in-process cache trades one truth for every instance against no network hop at all. Synchronous versus asynchronous trades an immediate answer against surviving a slow or dead dependency. Monolith versus microservices trades one deploy and cheap refactoring against independent deploys and independent scaling. Strong versus eventual consistency trades reading your own write against staying available when the network splits. Fan-out on write versus fan-out on read trades cheap reads against cheap writes. Knowing them cold matters because you can then move straight to the part that is actually specific to your system — which side of the axis this workload sits on and why — rather than re-deriving the axis in the room. Each has a sensible default: the boring, cheaper-to-operate side. Postgres, local cache only where staleness is safe, synchronous, monolith, strong consistency, fan-out on read. The skill is naming what has to be true to leave the default, and the honest answer is usually a number — a write rate, a fan-out size, a latency target — not a preference.

Think of it as

Six dials, not six switches. Every design turns each dial somewhere, including the designs that never discussed it — the default position is still a position. The work is knowing which way each dial trades and what measurement would justify moving it, so that "we are on the left of this one" is a statement about the workload rather than about the team's habits.

text

Six dials, and the number that moves each one

  Postgres ....... DynamoDB      write rate vs one primary's ceiling
  local cache .... shared Redis  seconds of staleness you can accept
  sync ........... async         caller's patience vs work duration
  monolith ....... services      deploy cadence + on-call ownership
  strong ......... eventual      does the user read their own write
  fan-out read ... fan-out write reads per write, and max followers

Naming the number is the design work.
Naming the side is just a preference.

What we're doing: Take one axis to a decision, using the four-part form and a real number.

fan-out-decision.txttext

The system: a follow-based feed.
  1.2M users, median 180 followers,
  largest account 2.4M followers,
  ~40 posts/second at peak,
  ~9,000 feed reads/second at peak.

The axis: fan-out on write vs on read.

FAN-OUT ON WRITE
  On each post, insert one row per
  follower into their timeline.
  Median post: 180 inserts. Fine.
  Largest account: 2.4M inserts for one
  post. At 40 posts/s overall, one
  celebrity post is 60,000x the median
  write and stalls the whole pipeline.

FAN-OUT ON READ
  On each feed request, query the posts
  of everyone you follow and merge.
  Median read: 180 lookups, cached.
  9,000 reads/s x 180 = 1.6M lookups/s.
  Too much to serve from the primary.

THE DECISION (hybrid)

BENEFIT
  Fan-out on write for accounts under
  10,000 followers covers 99.9% of
  posts and makes those reads a single
  range scan. Celebrity posts are
  merged in at read time, so no post
  ever produces more than 10,000
  writes.

COST
  Two code paths and a merge step. A
  follower of a celebrity gets their
  timeline assembled from two sources,
  which makes ordering and pagination
  harder than either pure design.

ALTERNATIVE
  Pure fan-out on read, with an
  aggressive per-user timeline cache.

REASON
  The read volume (9,000/s) is the
  binding constraint, and the celebrity
  tail is what makes pure write fan-out
  unsafe. The hybrid pays a bounded
  cost on both sides. Pure read fan-out
  was rejected because a cache miss
  storm would put 1.6M lookups/s on the
  primary with no ceiling.

THE THRESHOLD
  10,000 followers. It is a number we
  can move, and moving it is a config
  change rather than a redesign.

16: This is the number that decides the axis. Median fan-out being small is irrelevant on its own — the tail is what determines whether write fan-out is safe.
23: The read side has a number too. Neither pure design is rejected on a feeling; each is rejected by an arithmetic result the other side does not have.
27: "99.9% of posts" is what makes the hybrid worth its complexity. If the split were 60/40 the two code paths would both be hot and the simpler pure design would win.
57: Naming the threshold as a tunable number, rather than baking it into the design, keeps the decision reversible at the cost of one config value.

Why this works: The axis was known before the meeting started; nothing about fan-out was derived here. What the design work actually produced is three numbers — 2.4M maximum followers, 9,000 reads per second, and a 10,000-follower threshold — and those numbers, not a preference for one pattern, are what chose the shape.

Choosing a side before measuring the workload

Wrong

text

# "Feeds are fan-out on write. That's how
#  Twitter does it."
# Adopted at 1.2M users with a 2.4M-follower
# account in the dataset. One post now
# writes 2.4M rows and everything behind it
# in the queue waits.

Better

text

# Measure first:
#   max fan-out ......... 2.4M
#   reads per second .... 9,000
#   posts per second .... 40
# Then pick. The numbers rule out both pure
# designs and describe the hybrid for you.

What you see: A design that works in staging and in the first months of production, then degrades sharply the day a single popular account is added — with no gradual warning, because the failure is proportional to one account's follower count and not to overall traffic.

Why: These axes have no default winner; they have a winner per workload, and the deciding input is usually a distribution rather than an average. Copying a well-known system copies its answer without its numbers, and the numbers are the entire reason its answer was right.

Six axes, and what each one trades

Each row is a dial. Both ends are correct designs; the workload decides where it sits.

The six axes: what each side buys, and what you are really trading

The six axes: what each side buys, and what you are really trading
Decision	Left side buys	Right side buys	The real trade
PostgreSQL vs DynamoDB	Joins, ad-hoc queries, transactions across tables	Predictable latency and scale-out on a known key	Query flexibility for access-pattern discipline
Shared Redis vs local cache	One value every instance agrees on; invalidation works	No network hop; nothing to operate	Correctness of invalidation for a saved round trip
Synchronous vs asynchronous	An answer in the response; simple to reason about	The request survives a slow or dead dependency	Immediacy for availability, paid in eventual consistency
Monolith vs microservices	One deploy, cheap refactoring, local transactions	Independent deploys, scaling and ownership	Simplicity for team and scaling independence
Strong vs eventual consistency	Read-your-write; invariants enforced in one place	Availability during partitions and lower read latency	Certainty for availability and latency
Fan-out on write vs on read	Cheap reads: the timeline is already built	Cheap writes: nothing is built until asked for	Write amplification for read latency

The default, and what has to be true to leave it

The default, and what has to be true to leave it
Decision	Default	Leave it when
PostgreSQL vs DynamoDB	PostgreSQL	Access patterns are few and fixed, and the write rate or dataset outgrows one primary
Shared Redis vs local cache	Local cache for data that may be seconds stale	The value must be invalidated on demand, or all instances must agree
Synchronous vs asynchronous	Synchronous	The work is slower than the caller will wait, or must survive the dependency being down
Monolith vs microservices	Modular monolith	Two parts genuinely need different deploy cadence, scaling profile or on-call ownership
Strong vs eventual consistency	Strong	The workflow tolerates a stale read, and you can show the user what state it is in
Fan-out on write vs on read	Fan-out on read	Reads massively outnumber writes and fan-out per write stays bounded

Remember: Six axes, each with a default and an exit condition: Postgres → DynamoDB when access patterns are fixed and one primary is outgrown; local cache → shared Redis when invalidation must work; sync → async when the work outlasts the caller's patience; modular monolith → microservices when deploy cadence, scaling or on-call genuinely diverge; strong → eventual when the workflow tolerates staleness and can show its state; fan-out on read → on write when reads dominate and fan-out stays bounded. Know the axes cold so the design work is naming the number that moves the dial.