beginnerintermediateadvanced

System Design · Section 106

What a 5-Year Engineer Should Be Able to Explain

Level: advanced
Read: 16 min
Concepts: 1

Twelve questions, in four families. Sizing and choosing: how to estimate peak requests per second, how to choose between SQL and NoSQL, how to choose a cache strategy. Making reads fast: how to scale a read-heavy system, how to handle database contention, how to detect N+1 queries and network bottlenecks. Doing it exactly once: how to prevent double processing, how to handle retries, how to design idempotent APIs — which are one problem approached from three directions, and treating them as three separate topics is how a system ends up idempotent in two places and still double-charging. Surviving failure: how to model an asynchronous workflow, how to design for regional failure, how to recover from a dependency failure. What makes the list worth having is not the topics but the standard for an answer. Every one of the twelve can be answered with a product name, and none of those answers survives a single follow-up: "we use SQS" is cited to prevent double processing by systems whose queue is explicitly at-least-once, and "we use Redis" is a technology where a cache strategy was asked for. A real answer names a mechanism, states one assumption, and fits in three sentences — and the assumption is the part a follow-up question is aiming at, which is why it has to be said out loud. If an answer takes longer than three sentences it is being derived rather than recalled, and having these assembled is what lets a design conversation stay on the part that is actually specific to the system in front of you.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Four families: sizing and choosing, making reads fast, doing it exactly once, surviving failure.
A product name is a possible implementation, not an answer — it skips the step where the problem gets characterised.
Double processing, retries and idempotent APIs are one problem; one key must travel the whole path.
Every strong answer states one assumption, and the follow-up question always targets it.
An answer longer than three sentences is being derived, which is what the list actually measures.

What you will be able to do

Answer any of the twelve in three sentences, naming a mechanism and one stated assumption
Estimate peak RPS from a business number and say which peak factor you applied and why
Recognise a product-name non-answer, including one that cites a guarantee the product does not make
Carry a single idempotency key across API, retry and consumer boundaries instead of guarding each separately

The twelve

Four families, twelve questions, and the standard that separates an answer from a technology name.

Twelve questions, and what an answer looks like

coreadvanced

Twelve questions, in four families. Sizing and choosing: how to estimate peak requests per second, how to choose between SQL and NoSQL, how to choose a cache strategy. Making reads fast: how to scale a read-heavy system, how to handle database contention, how to detect N+1 queries and network bottlenecks. Doing it exactly once: how to prevent double processing, how to handle retries, how to design idempotent APIs. Surviving failure: how to model an asynchronous workflow, how to design for regional failure, how to recover from a dependency failure. What makes the list useful is not the topics — it is that every one of them can be answered with a product name, and none of those answers survives a single follow-up. "We use Redis" is not a cache strategy; the strategy is which pattern, keyed on what, invalidated how, and what happens on a miss storm. "We use SQS" is not a way to prevent double processing; delivery is at-least-once, so the consumer has to be idempotent regardless. A real answer names a mechanism and the condition under which it stops working, and it can usually be given in three sentences. If it takes longer, the answer is being reconstructed rather than recalled — which is the actual thing the list measures.

Think of it as

Twelve questions a colleague could ask you at a whiteboard with no warning. The test is not whether you could work each one out — you could — it is whether the answer is already assembled. Assembled answers are what let a design conversation stay on the part that is specific to this system, instead of re-deriving standard machinery in the room.

text

Four families, twelve questions

  SIZING AND CHOOSING
    peak RPS · SQL vs NoSQL · cache strategy

  MAKING READS FAST
    scale a read-heavy system · database
    contention · N+1 and network bottlenecks

  DOING IT EXACTLY ONCE
    prevent double processing · retries ·
    idempotent APIs
    (one problem from three directions)

  SURVIVING FAILURE
    async workflow · regional failure ·
    dependency failure

Three sentences each. Longer means you are
deriving it, not recalling it.

What we're doing: Answer two of the twelve out loud, and see where the follow-up lands.

answering.txttext

Q1. "How would you estimate peak RPS for
     this?"

WEAK
  "It depends on traffic."
  True and empty. It ends the conversation
  rather than starting it.

STRONG
  "Start from the business number. Say
   2 million orders a day. That is 2e6 /
   86,400 = ~23 orders/sec average. Order
   creation is maybe 1 in 20 requests, so
   ~460 rps average overall. Traffic is not
   flat -- most consumer products peak at
   3-10x average, so I would size for
   ~2,000-4,600 rps and say out loud that
   I used a 5x peak factor. If the product
   has a daily batch or a marketing send,
   the factor is higher and I would ask
   for the real hourly distribution."

  Three numbers, one stated assumption, and
  the assumption is the part someone can
  correct.

FOLLOW-UP
  "What if it is a ticketing site?"
  -> "Then the peak factor is useless. Sales
      are scheduled, so peak is not a
      multiple of average -- it is however
      many people were waiting for 10am. I
      would size for the queue at the door,
      not for the daily total."

  This is why the assumption has to be
  spoken. The follow-up attacks the
  assumption, and there is nothing to attack
  if it was never stated.

---

Q2. "How do you prevent double processing?"

WEAK
  "The queue handles it -- we use SQS."

  Follow-up: "SQS standard queues are
  at-least-once. What happens when a message
  is delivered twice?"
  -> The answer was a product name, and the
     product does not make the guarantee it
     was cited for.

STRONG
  "Delivery is at-least-once, so the
   consumer has to be idempotent -- I would
   not rely on the broker. Concretely: the
   handler writes the effect and the message
   id in the same transaction, with a unique
   constraint on the id. A redelivery hits
   the constraint, the handler treats that
   as success and acknowledges. That covers
   duplicates from redelivery, from a
   producer retry, and from a replay."

FOLLOW-UP
  "What if the effect is a call to a payment
   provider, not a database write?"
  -> "Then the dedupe key has to reach the
      provider too -- their idempotency key.
      The local record alone cannot help,
      because the side effect happens
      outside my transaction."

  The follow-up moves the effect outside the
  database, which is where the mechanism
  actually gets tested.

12: Working forward from a business number rather than from an infrastructure guess is what makes the estimate checkable. Every step is one arithmetic operation someone can dispute.
20: Naming the peak factor is the whole answer. An unstated 5x and an unstated 20x produce very different systems and look identical in a design document.
48: This is the most common product-name non-answer in the whole list, and it is wrong on the facts: standard queues are explicitly at-least-once, so the guarantee being cited does not exist.
68: Moving the effect outside the database is the standard follow-up for any idempotency answer, and it is where a memorised answer separates from an understood one.

Why this works: Every strong answer here is three or four sentences and contains one stated assumption. That assumption is what the follow-up targets — which is the point: an answer with no assumption in it cannot be examined, and an answer that has never been examined is the one that fails in a design nobody questioned.

Answering with a product name

Wrong

text

"How do you scale reads?"    "Redis."
"SQL or NoSQL?"              "Dynamo."
"Regional failure?"          "Multi-region."
"Retries?"                   "Celery does it."
# Four answers, zero mechanisms. Each names
# a thing that could implement the answer
# and none of them says what the answer is.

Better

text

"How do you scale reads?"
  "Find out whether it is a hot-key problem
   or a volume problem first. Hot keys ->
   cache with a per-key lock. Volume ->
   read replicas, and then decide per
   workflow which reads can tolerate lag.
   Redis and replicas are how; the decision
   is which one the traffic actually needs."

What you see: A design conversation where every question is answered instantly, nothing is disputed, and the resulting system has three technologies in it that nobody can say what problem they solve.

Why: A product name is a possible implementation of an answer, not the answer, so it skips the step where the problem gets characterised. That step is the only part that is specific to your system — and it is exactly the part a follow-up question is aiming at.

Twelve questions in four families

Sizing and choosing

Peak RPS

average × a stated peak factor

SQL vs NoSQL

by access patterns

Cache strategy

pattern + key + TTL + invalidation

Making reads fast

Scale a read-heavy system

Database contention

N+1 and network bottlenecks

count queries in a trace

Doing it exactly once

Prevent double processing

Retries

bounded, jittered, transient only

Idempotent APIs

one problem, three directions

Surviving failure

Async workflows

a state machine with terminal states

Regional failure

RPO and RTO decide the shape

Dependency failure

timeout · breaker · degraded mode

The twelve, with an answer that names a mechanism

The twelve, with an answer that names a mechanism
Family	Question	The answer in one line
Sizing and choosing	How do you estimate peak RPS?	Daily actions ÷ 86,400 for the average, then × 3 to 10 for peak — and say which factor you used and why
Sizing and choosing	How do you choose SQL vs NoSQL?	By access patterns: unknown or relational queries → SQL; few fixed patterns on a known key plus a scale-out need → NoSQL
Sizing and choosing	How do you choose a cache strategy?	Read-through for read-heavy lookups, write-through when staleness is unacceptable, write-behind only when loss is survivable — plus a TTL and an invalidation rule
Making reads fast	How do you scale a read-heavy system?	Cache the hot keys, add read replicas for the rest, and decide per workflow which reads tolerate replica lag
Making reads fast	How do you handle database contention?	Shorten transactions, order lock acquisition consistently, use conditional updates instead of read-then-write, and shard the hot row when one row is the bottleneck
Making reads fast	How do you detect N+1 and network bottlenecks?	Count queries per request in a trace, not by reading code — an N+1 shows as request duration rising linearly with result-set size
Doing it exactly once	How do you prevent double processing?	A deduplication key stored with the effect, checked in the same transaction that applies it
Doing it exactly once	How do you handle retries?	Bounded attempts, exponential backoff with jitter, retry only transient errors, then dead-letter with an alert
Doing it exactly once	How do you design idempotent APIs?	Client-supplied key on every unsafe write; store key → response; the second call returns the first result rather than repeating the effect
Surviving failure	How do you model an asynchronous workflow?	An explicit state machine with a durable record per step, a terminal state for every path, and a status the user can see
Surviving failure	How do you design for regional failure?	Decide RPO and RTO first; they determine whether you need active-active, warm standby or a restore — not the other way round
Surviving failure	How do you recover from a dependency failure?	Timeout, circuit breaker, a defined degraded mode, and a queue for the work that can wait

The same question, answered two ways

The same question, answered two ways
Question	A product name (not an answer)	An answer
Cache strategy?	"We use Redis."	"Read-through on the product page, keyed by product id, 5-minute TTL, invalidated on write, with a per-key lock so a miss on a hot key does not stampede."
Prevent double processing?	"We use SQS."	"Delivery is at-least-once, so the consumer stores the message id with the effect in one transaction; a redelivery finds the row and returns."
Regional failure?	"We are multi-region."	"RPO 5 minutes, RTO 30 minutes. Async replication to a warm standby, DNS failover, and we tested the promotion in March."
Database contention?	"We use transactions."	"The order transaction is three statements and holds no external call. The stock row uses a conditional update, so the loser gets zero rows and a 409."

Remember: Twelve questions in four families: sizing and choosing (peak RPS, SQL vs NoSQL, cache strategy), making reads fast (read-heavy scaling, contention, N+1 and network bottlenecks), doing it exactly once (double processing, retries, idempotent APIs — one problem from three directions), and surviving failure (async workflows, regional failure, dependency failure). Each answer names a mechanism and one stated assumption, and fits in three sentences. A product name is not an answer, and the follow-up question is always aimed at the assumption you did not say out loud.