beginnerintermediateadvanced

System Design · Section 105

Practical Projects

Level: advanced
Read: 55 min
Concepts: 8

Eight projects, each small enough to actually finish, each chosen because it forces one lesson the others cannot. A URL shortener teaches read/write asymmetry — redirects outnumber creates by orders of magnitude, so the two paths get designed separately — and makes you choose an ID scheme by the property you want rather than by code length. A notification platform teaches the intent/delivery split: one accepted request fans out into several attempts across several channels, each with its own state and its own provider, which means two records rather than one status column. A file processing platform teaches that bytes never touch your application tier, and that a worker holding a lease it extends is what turns a crash into lost work instead of a stuck job. An e-commerce order system is the hardest and the most valuable: it is the smallest realistic place where money, inventory and a third party must agree, so a database transaction may never contain a provider call, and the gap between "charged" and "recorded" needs reconciliation rather than optimism. A chat platform is the only one whose connections are stateful, which forces pub/sub between instances, a per-room sequence instead of a clock, and a client-held cursor instead of a server-side delivery buffer. A multi-tenant SaaS platform teaches that isolation is a property that must hold on every path — and that the leaks are on the paths with no request context: cache keys, background jobs, exports, search indexes and logs. A search platform teaches that a derived store must be rebuildable, which makes reindexing behind an alias routine and disaster recovery the same procedure. And a distributed rate limiter, the smallest of the eight, teaches atomicity: read-compare-write overshoots the limit by roughly the number of servers, and the fix is to move the whole decision into one operation the store performs indivisibly. That last bug is the same shape as the inventory race in project 4, which is the strongest argument for building both.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Each project is chosen for one lesson: asymmetry, intent vs delivery, leases, transaction boundaries, stateful connections, isolation, derived stores, atomicity.
Two of them teach the same race from different angles — the last unit of stock and the last token in the bucket.
Build the boring version first: REST before WebSockets, SQL LIKE before an index, one server before twelve.
A definition of done is a failure you cause on purpose: kill a worker, kill an instance, delete the index.
The mechanics live in the sections each project links to; the project is where you find out which parts you only thought you understood.

What you will be able to do

Choose a project by the lesson you are missing rather than by how interesting it sounds
Design each system as a build order where every step is a working system, not a stage of one
Recognise the read-then-write race in two unrelated-looking places and fix both with one mechanism
State a definition of done as a deliberate failure the system survives, and then actually cause it

Start here

Two projects whose lessons — read/write asymmetry, and separating intent from delivery — shape everything after them.

Project 1 — URL shortener

coreintermediate

Build a service that turns a long URL into a short code and redirects that code back to the original. It looks small, and that is why it is the first project: the whole thing fits in a weekend, but it forces four decisions that recur everywhere. The first is the read/write asymmetry — redirects outnumber creations by a factor of hundreds or thousands, so the create path and the redirect path get designed separately rather than as two endpoints on the same service. The second is ID generation, which has no obviously correct answer until you name the property you want: a random code is unguessable but needs a collision check, a counter in base62 is short and dense but leaks how many links exist and needs coordination between instances, and a hash of the URL deduplicates identical links but makes two users share one code and therefore one set of analytics. The third is caching, because the redirect path is a pure key lookup and belongs in Redis — with the database still the system of record, so a cache flush costs latency rather than data. The fourth is that analytics must not be on the redirect path: a click writes an event to a queue and the redirect returns, because a redirect that waits on an analytics write has coupled its availability to a system nobody would page for.

Think of it as

Two products sharing one database. A write product that is low volume, needs validation, rate limiting and abuse checks, and can afford to be slow; and a read product that is high volume, does one key lookup, and must be fast and boring. Designing them as one service is what makes people put the click counter in the redirect handler.

text

Build order — each step works before the next

  1  POST /links + GET /{code}, Postgres only
  2  ID generation: pick a scheme, write down
     which property you chose it for
  3  Redis read-through on the redirect path
  4  Click events -> queue -> analytics store
  5  Rate limit on POST /links (per API key)
  6  Expiry: TTL column + a sweep, 410 on
     an expired code

Definition of done: you can state your p99
redirect latency, what happens when Redis is
down, and why you chose 301 or 302.

What we're doing: Get the redirect path right — the part that is one line of code and three decisions.

redirect-path.txttext

THE NAIVE HANDLER

  GET /{code}
    row = db.query(code)          # 8ms
    db.execute(insert click_row)  # 6ms
    return 302 -> row.url
  p99 ~ 40ms, and every redirect
  writes to the database.

THREE DECISIONS IT SKIPPED

1. WHERE THE LOOKUP COMES FROM
   Redis read-through: hit is ~1ms.
   A miss reads Postgres and fills the
   cache. Redis down -> every request
   falls through to Postgres. That is
   slower, not broken -- but only if
   Postgres can carry the full rate.
   Size for that, or shed load.

2. WHAT THE CLICK WRITE COSTS
   The insert is on the critical path,
   so redirect availability is now
   analytics availability. Push a click
   event to a queue instead and return.
   Losing a few click events is fine;
   losing redirects is not.

3. 301 OR 302
   301 (permanent) lets browsers and
   proxies cache the redirect forever.
   Fewer requests reach you -- and your
   click count silently stops counting
   repeat visitors, and you can never
   change or delete that link for
   anyone who cached it.
   302 (found) keeps every click, keeps
   the link editable, and costs traffic.
   Analytics is usually the product, so
   302 usually wins. Either is correct;
   choosing by accident is not.

THE HANDLER AFTER

  GET /{code}
    url = cache.get(code)
          or db_then_fill(code)   # ~1ms
    queue.publish(click_event)    # async
    return 302 -> url

12: A read-through cache changes the failure mode as well as the latency: Redis being down must degrade to slow, never to wrong. That only holds because Postgres remains the record.
22: This is the general form of the rule, not a URL-shortener detail: a critical path may not synchronously depend on a system whose failure you would not page for.
31: The 301 trade is the one people discover in production, months later, when the click graph flattens and nobody can explain why. It is a caching decision disguised as a status code.

Why this works: Nothing in the redirect path is difficult, which is exactly what makes it a good first project: every one of the three decisions has a defensible answer in both directions, and getting them wrong produces a system that works perfectly in testing and behaves strangely at scale.

Choosing the ID scheme by its length

Wrong

text

# "Six characters is nicer than eight."
# -> random 6-char base62 = 56.8 billion
#    codes, but collision probability rises
#    with the square of how many exist, so
#    inserts start retrying long before the
#    space is anywhere near full.
# And nobody wrote down whether codes are
# meant to be unguessable.

Better

text

# Name the property first:
#   "codes must not be enumerable"
#     -> random, with a unique constraint
#        and a retry on conflict
#   "codes must be as short as possible"
#     -> counter in base62, accepting that
#        it leaks link volume
# Length falls out of the choice.

What you see: Either an insert path that quietly retries more and more often as the table grows, or a competitor who can count your total links by shortening one URL a day.

Why: Code length is a consequence of the scheme, and the scheme is a consequence of one property — unguessable, shortest, or deduplicating. Picking the length first leaves the property unchosen, so it gets decided by whichever scheme happened to fit the length.

URL shortener — the two paths

Client

Create API

POST /links · rate limited

ID generation

base62 · scheme is a choice

PostgreSQL

system of record

Redirect service

GET /{code} → 302

Redis

code → URL, read-through

Click events

fire and forget

Analytics store

Three ID schemes, and the property each one buys

Three ID schemes, and the property each one buys
Scheme	Buys	Costs	Pick it when
Random base62, 7 chars	Unguessable; no coordination between instances	Needs a uniqueness check and a retry on collision	Links may be private, and enumeration would be a leak
Counter → base62	Shortest possible codes; no collision check at all	Leaks total link count; needs a coordinated counter or per-instance blocks	Codes are public anyway and shortness is the product
Hash of the URL	Identical URLs collapse to one code, saving storage	Two users share a code, so they share analytics and expiry	Dedup is genuinely wanted and per-user tracking is not

The two paths, sized separately

The two paths, sized separately
Path	Shape	What it needs	What it must not do
POST /links (create)	Low volume, validated, authenticated	Rate limiting, URL validation, abuse checks, an ID	Assume the caller is friendly
GET /{code} (redirect)	High volume, one key lookup	Cache read-through, a 301 or 302, sub-10 ms p99	Write to the analytics store synchronously

Remember: A URL shortener is two products sharing a database: a low-volume create path that needs validation and rate limiting, and a very high-volume redirect path that is one cached key lookup. Choose the ID scheme by the property you want (unguessable / shortest / deduplicating), keep Postgres as the record with Redis read-through so a flush costs latency and not data, push click events to a queue so redirect availability never depends on analytics, and choose 301 versus 302 deliberately — 301 is cached forever and silently ends your click counting.

Project 2 — Notification platform

coreintermediate

Build a service other services call to send email, SMS and push. It teaches the intent/delivery split better than anything else, because the split is unavoidable here: one accepted request ("tell user 91 their order shipped") fans out into several attempts across several channels, each with its own state, its own provider and its own failure. So there are two tables, not one. A notification row records what was asked for and is created inside the caller's request. Delivery rows record each attempt on each channel and are created by workers afterwards. Separate worker pools per channel matter more than they look: email providers rate-limit differently from SMS providers, and one pool means a backlog of SMS retries starves email. Deduplication is the other unavoidable piece — callers retry, and a retried "order shipped" must not send a second email, so the caller supplies a key and the same key returns the same notification. Templates belong on the platform rather than in each caller so that a wording fix is one deploy. And delivery tracking is what makes the whole thing operable: without a per-attempt state you cannot answer "did user 91 get it", which is the only question anyone ever asks.

Think of it as

A post office. You hand over a letter and get a receipt; that is the intent, and it is accepted or rejected immediately. What happens next — sorting, routing, a failed delivery, a second attempt, a return to sender — is a separate process with its own record, and the receipt number is how you ask about it later. Nobody expects the counter clerk to wait while the letter is delivered.

text

Build order — each step works before the next

  1  POST /notifications -> row + 202 + id
     GET /notifications/{id} -> status
  2  Queue + one email worker. Provider call,
     store the provider message id.
  3  Delivery rows, one per channel attempt.
  4  Retries: 5 attempts, backoff + jitter,
     then dead-letter. 429 slows the pool.
  5  Dedupe key on POST, unique per caller.
  6  Templates, versioned, rendered by the
     worker so a fix is one deploy.
  7  SMS + push as separate pools.
  8  Provider webhooks -> delivery status.

Definition of done: you can answer "did user
91 get the shipping email, and if not, why"
from stored state alone.

What we're doing: Answer the only question anyone asks — "did user 91 get it?" — and see what state that requires.

delivery-tracking.txttext

THE DESIGN THAT CANNOT ANSWER IT

  notifications
    id, user_id, event, status
  status is one of: pending, sent, failed

  Support asks: "did user 91 get the
  shipping email?"
  The row says status = sent.
  Sent on which channel? Email and push
  were both chosen. Sent, meaning the
  provider took it, or meaning it arrived?
  Attempted how many times? Which address?
  The row cannot say. One status column is
  covering three different questions.

THE DESIGN THAT CAN

  notifications
    id, user_id, event, dedupe_key,
    payload, created_at
  deliveries
    id, notification_id, channel, provider,
    provider_message_id, attempt,
    state, error, updated_at

  Now:
    email   attempt 1  provider_error 451
    email   attempt 2  sent      msg_a91f
    email   --         bounced   mailbox full
    push    attempt 1  delivered

  Answer: "No. The email bounced -- their
  mailbox is full -- but the push notification
  was delivered at 14:02."

WHY THE BOUNCE MATTERS

  A bounce is permanent. Retrying it wastes
  attempts and damages sender reputation
  with the provider, which degrades delivery
  for every other user. So bounced is a
  terminal state that also writes back to
  the user's address record.

WHAT DEDUPE PROTECTS

  The shipping service times out and retries
  POST /notifications with the same
  dedupe_key. Without it: two emails. With
  it: the API returns the original
  notification id and enqueues nothing.

13: One status column across multiple channels is the single most common shape error in this project. The moment two channels exist, status is a property of an attempt, not of the intent.
30: The attempt history is what turns a support question into a stored fact. Note that it also records a provider error code — paraphrasing it would lose the only detail that explains the retry.
44: Treating a bounce as a retryable failure is how a notification platform degrades delivery for every user at once, via a shared sender reputation nobody is monitoring.

Why this works: Delivery tracking looks like a reporting feature and is actually the data model. Once you can answer "did they get it, on which channel, after how many attempts, and why not", every other requirement — retries, dedupe, preferences, provider limits — has somewhere to live.

Treating a provider 429 as a failed send

Wrong

text

# Worker: any non-2xx -> record failure,
# schedule a retry.
# The provider returns 429 (rate limited).
# The worker retries. So do the other 200
# messages in flight. All of them are now
# retrying into a limit that was already
# exceeded, and the backlog grows.

Better

text

# 429 is not a message failure -- it is a
# signal about the pool.
#   - honour Retry-After
#   - slow the whole channel pool, not just
#     this message
#   - do not count it as an attempt
# The backlog drains at the provider's rate
# instead of fighting it.

What you see: A queue that grows fastest exactly when the provider is busiest, and per-message attempt counts exhausted by errors that never actually attempted anything.

Why: A rate-limit response is feedback about throughput, and the correct reaction is backpressure on the pool rather than a per-message retry. Handling it per message multiplies the load on the limit that produced it, which is the same amplification that turns a slow dependency into an outage everywhere else.

Notification platform — intent in, deliveries out

Calling service

Notify API

dedupe key → 202 + id

Notifications

intent, one row

Work queue

one topic per channel

Email pool

own rate limit

SMS pool

own rate limit

Providers

2xx = accepted, not delivered

Delivery rows

per attempt, per channel

Dead letter

after 5 attempts

The two records, and why they cannot be one row

The two records, and why they cannot be one row
Record	Created by	Holds	Lifecycle
Notification (intent)	The API, inside the caller's request	user, event, payload, dedupe key, channels chosen	accepted → resolved (all channels terminal)
Delivery (attempt)	A worker, per channel	channel, provider, provider message id, attempt number, error	queued → sent → delivered / bounced / failed

What a provider actually tells you, and when

What a provider actually tells you, and when
Signal	When it arrives	What it means	What it does not mean
2xx from the send call	Immediately	The provider accepted the message	The user received it
429 from the send call	Immediately	You exceeded the provider's rate limit	The message failed — it was never attempted
Delivery webhook	Seconds to hours later	The provider believes it reached the device or inbox	The user read it
Bounce webhook	Seconds to days later	Permanent failure — stop retrying this address	A transient error worth a retry

Remember: Two records: a notification is the intent, created synchronously with a caller-supplied dedupe key; a delivery is one attempt on one channel, created by a worker. Separate worker pools per channel so one provider's backlog cannot starve another. A provider 2xx means accepted, not delivered — real status arrives later by webhook, and a bounce is terminal. Treat 429 as backpressure on the pool rather than a per-message retry, and re-check preferences and quiet hours at send time, because a queue is a decision made in the past.

Async work and correctness

Long-running work, leases, and the smallest system where money and inventory have to agree.

Project 3 — File processing platform

coreintermediate

Build something that accepts a large file, does slow work on it, and gives the result back — resizing images, transcoding video, extracting text from a PDF. The central lesson is that the bytes never touch your application servers. The client asks the API for a signed upload URL, uploads directly to object storage, then tells the API the upload finished; the API stores metadata and enqueues a job. Doing it the other way, streaming the file through your API, ties every upload to one server's memory, bandwidth and deploy cycle, and a rolling restart kills every upload in flight. The second lesson is that progress tracking is a requirement, not a nicety: work measured in minutes needs a job record with a state, a percentage and an error field, because a client that cannot poll will either retry (doubling the work) or hang. The third is the multipart upload — a 4 GB file over a phone connection will fail partway, and multipart lets it resume from the failed part rather than from zero. And the whole thing has an ownership rule worth stating: object storage holds bytes, the database holds truth. A file that exists in storage with no row is garbage to be swept; a row with no object is a failed upload to be reported.

Think of it as

A dry cleaner. You hand the item over at the counter and get a ticket; the counter does not clean anything. The work happens in the back at its own pace, the ticket tells you what stage it is at, and collection is a separate visit with the ticket. The counter staying free is the entire reason the shop can take the next customer.

text

Build order — each step works before the next

  1  POST /jobs -> job row + signed PUT URL
     Client PUTs the object directly.
  2  POST /jobs/{id}/complete -> validate the
     object exists, size + type, then enqueue
  3  Worker: claim with a lease, process,
     write output object, update job
  4  GET /jobs/{id} -> state + percent
  5  Signed GET URL for the result, short TTL
  6  Multipart upload for files over ~100 MB
  7  Sweep: objects with no row, rows stuck
     in awaiting_upload past their URL expiry

Definition of done: kill a worker mid-job and
the job completes anyway, exactly once.

What we're doing: Survive a worker dying halfway through a 40-minute transcode, without producing two outputs.

worker-crash.txttext

THE SETUP

  Job 8812: transcode a 3.1 GB video.
  Expected duration ~40 minutes.
  Worker claims it, gets to 60%, and the
  instance is terminated by a scale-in.

WHAT A NAIVE DESIGN DOES

  The message was deleted from the queue
  when the worker picked it up.
  The job row still says processing, 60%.
  Nothing will ever move it again.
  Support finds it three days later.

WHAT A LEASE DOES

  The worker does not delete the message.
  It holds a lease -- a visibility timeout
  the worker extends while it works, e.g.
  every 60s for 5 more minutes.
  Worker dies -> no more extensions ->
  the lease expires -> the message becomes
  visible -> another worker claims it.

  Now the crash costs 60% of one transcode,
  not a stuck job.

WHY THE OUTPUT KEY MUST BE DETERMINISTIC

  The second worker reprocesses from the
  start and writes its output. If the key
  were random -- outputs/{uuid}.mp4 -- the
  first worker's partial output would
  linger forever, unreferenced and unswept,
  and a third attempt would leave two.

  Use outputs/{job_id}/720p.mp4. Rerunning
  overwrites the same key. The work is not
  idempotent, but its effect is.

WHAT PROGRESS MUST NOT BE

  percent must be written by the worker
  that currently holds the lease, keyed by
  attempt. Otherwise a slow zombie worker
  -- one that lost its lease but is still
  running -- keeps writing 62%, 63% over
  the new worker's progress.

THE REMAINING HOLE

  The zombie can also still write its
  output object. A deterministic key means
  it overwrites with a valid result, which
  is harmless here. For work where it would
  not be harmless, the write needs a
  conditional check on the job's current
  attempt number.

9: Deleting the message on claim is the default in most naive queue code, and it converts every worker crash into a permanently stuck job with no alert attached to it.
20: Lease extension is what makes long jobs safe on a queue whose visibility timeout is measured in minutes. Without it, either the timeout is too short for the job or too long to recover from a crash.
33: A deterministic output key is the cheapest way to make a non-idempotent operation have an idempotent effect. It is worth reaching for before any coordination mechanism.
48: The zombie-worker case is the one people skip. It is not hypothetical — a paused VM, a long GC or a network partition all produce a worker that believes it still holds a lease it has lost.

Why this works: Everything in this project is easy until a worker dies, and a worker dying is routine — scale-in, a deploy, a spot instance reclaim. Leases, deterministic keys and attempt-scoped progress are the three mechanisms that turn "a crash breaks a job" into "a crash costs some work".

Loading the file into memory to process it

Wrong

text

data = storage.get(key).read()   # 3.1 GB
result = transcode(data)
storage.put(out_key, result)
# One job needs 6+ GB of RAM. Two concurrent
# jobs kill the worker. Job size, not job
# count, decides how many fit -- so capacity
# planning becomes impossible.

Better

text

with storage.stream(key) as src, \
     storage.multipart(out_key) as dst:
    for chunk in transcode_stream(src):
        dst.write(chunk)
# Memory is bounded by chunk size, not file
# size. Worker capacity becomes a number of
# concurrent jobs, which you can plan for.

What you see: Workers that are fine for weeks and then die in pairs whenever two large files arrive together — with an out-of-memory kill that leaves no application log to explain it.

Why: Reading a whole object into memory makes peak memory a function of the largest input a user can supply, which is not a number you control. Streaming makes it a function of chunk size, which you do, and that is the difference between a worker pool you can size and one that fails by input.

File processing — bytes bypass the API entirely

Client

Job API

metadata only — never bytes

Jobs table

state · percent · error

Object storage

signed PUT, then signed GET

Job queue

lease + visibility timeout

Worker pool

idempotent, streams the object

Output object

deterministic key

Two upload designs, and what each one costs

Two upload designs, and what each one costs
Design	Path the bytes take	What breaks	When it is acceptable
Through the API	client → app server → object storage	Server memory and bandwidth scale with file size; a deploy kills uploads in flight	Small files only, where you must inspect content before storing
Signed URL (direct)	client → object storage	You cannot inspect the file until after it lands	Almost always — validate after upload, before enqueueing the job

The job record — the states a client can be shown

The job record — the states a client can be shown
State	Set by	Means	Client shows
awaiting_upload	API, on job creation	Signed URL issued; no object yet	Upload progress from the browser
queued	API, when the client confirms upload	Object exists and was validated	Waiting to start
processing	Worker, on claim	A worker holds a lease on this job	Percent complete
succeeded	Worker	Output written; result key recorded	A signed download URL
failed	Worker, after retries exhausted	Terminal; error is stored verbatim	The reason, and whether retrying will help

Remember: The bytes never touch your API. Client asks for a signed PUT URL, uploads straight to object storage, then confirms; the API stores metadata and enqueues. Object storage holds bytes, the database holds truth — an object with no row is garbage, a row with no object is a failed upload. Workers claim with a lease they extend, stream rather than load, and write to a deterministic output key so a retry overwrites instead of duplicating. Progress is a job field a client can poll; a client that cannot poll will retry, and doubling minutes of work is expensive.

Project 4 — E-commerce order system

coreadvanced

Build checkout: reserve stock, take payment, create an order, then fulfil it. This is the hardest of the eight projects and the one worth the most, because it is the smallest realistic system where money, inventory and a third party all have to agree. Three things make it hard. First, the payment provider is outside your database, so no transaction can cover both the charge and the order row — the design has to decide what happens in the gap between them, in both directions. Second, inventory is contended: two customers want the last unit at the same moment, and the database, not the application, has to be the thing that decides. Third, every step is retried by somebody — the customer refreshes, the client times out and retries, the queue redelivers — so every write needs to be safe to run twice. The shape that handles all three is: one short local transaction that reserves stock and creates the order in `pending_payment`, then the payment attempt outside any transaction, then a second short transaction that records the result and writes an outbox row, then fulfilment consuming that outbox asynchronously. The rule underneath it is that a database transaction must never contain a network call to something you do not control.

Think of it as

A restaurant kitchen ticket. Taking the order, taking payment and cooking are three separate steps with a physical record between them, and the ticket exists precisely so that any one of them can fail without losing the others. The moment you try to make them one atomic act — no payment until the food is served, no cooking until the card clears — the restaurant stops working.

text

Build order — each step works before the next

  1  Orders + items. POST /orders with an
     Idempotency-Key; same key -> same order.
  2  Inventory: conditional decrement
       UPDATE stock SET qty = qty - 1
       WHERE sku = ? AND qty >= 1
     0 rows updated = out of stock. No read
     first. The database decides.
  3  Payment outside any transaction, with
     the provider's idempotency key = your
     payment attempt id.
  4  Outbox row written in the same
     transaction as the order state change.
  5  Relay publishes outbox -> fulfilment
     consumes, idempotently.
  6  Reservation expiry sweep: unpaid holds
     return stock after N minutes.
  7  Reconciliation job vs the provider.

Definition of done: kill the process between
every pair of steps; no money is lost and no
stock is double-sold.

What we're doing: Sell the last unit to exactly one of two simultaneous customers.

last-unit.txttext

THE SETUP

  sku SHOE-42, qty = 1.
  Two customers press Buy in the same
  millisecond, on two application servers.

WHAT READ-THEN-WRITE DOES

  Server A: SELECT qty -> 1. OK to sell.
  Server B: SELECT qty -> 1. OK to sell.
  Server A: UPDATE qty = 0
  Server B: UPDATE qty = 0
  Two orders, one shoe. Both customers are
  charged. The bug is not in either server;
  it is in the assumption that a read
  predicts the state at write time.

WHAT A CONDITIONAL UPDATE DOES

  UPDATE stock
     SET qty = qty - 1
   WHERE sku = 'SHOE-42' AND qty >= 1;

  Server A: 1 row updated -> proceed.
  Server B: 0 rows updated -> out of stock,
            return 409 before charging.

  There is no read. The condition and the
  write are the same statement, so the
  database orders them and one loses.

WHAT THE LOSER MUST SEE

  409 with a specific reason. Not a generic
  500, and not a silent "order created" that
  fails during fulfilment -- by then the
  customer has been charged, and a refund is
  a worse experience than a rejection.

WHY THE RESERVATION EXPIRES

  Customer A now holds the unit but has not
  paid. If they abandon checkout, the shoe
  is unsellable forever.
  So the reservation row carries held_until.
  A sweep returns stock for expired,
  unpaid reservations -- and the sweep must
  not race the payment: releasing a
  reservation whose payment is in flight
  recreates the double-sell from the other
  direction.
  Fix: the payment-record transaction
  re-checks the reservation is still held,
  and the sweep skips reservations with a
  payment attempt in progress.

WHAT THIS COSTS

  Contention on one row per SKU. For a
  normal catalogue that is nothing. For a
  flash sale of one SKU it is the entire
  bottleneck, and that is when you shard
  the counter or move to a queue-based
  allocation -- a different design, chosen
  for a measured reason.

14: This is the general lesson, not a stock-keeping detail: a value read in one statement is a fact about the past, and any decision based on it is a race unless the write re-checks it.
26: Zero rows updated is the signal. It costs nothing, needs no lock held across statements, and works identically across every instance of the application.
45: The expiry sweep racing the payment is the second-order bug that appears once the first is fixed. It is the reason the record transaction re-checks rather than assuming its reservation survived.
57: Naming when this design stops working — one hot SKU — is what keeps it a decision rather than a habit. The flash-sale case genuinely needs something else.

Why this works: Inventory is the clearest place to learn that correctness under concurrency belongs in the database, not in application code. The conditional update is three lines, needs no coordination service, and removes an entire class of bug that no amount of careful application logic can close.

Calling the payment provider inside the transaction

Wrong

text

BEGIN;
  UPDATE stock SET qty = qty - 1 ...;
  INSERT INTO orders ...;
  charge = provider.charge(card, amount);  # 400ms-30s
  UPDATE orders SET state = 'paid' ...;
COMMIT;
# The row lock on that SKU is held for the
# provider's latency. And on a timeout the
# transaction rolls back -- while the charge
# may well have succeeded.

Better

text

BEGIN; reserve stock; insert order; COMMIT;
charge = provider.charge(..., idem_key)
BEGIN; record result; insert outbox; COMMIT;
# Locks are held for microseconds.
# A timeout leaves a recoverable state:
# retry with the same key, or let
# reconciliation resolve it.

What you see: Under load, lock waits on the hottest SKUs that track the payment provider's latency exactly — and, separately, customers charged for orders that do not exist because the transaction rolled back after a provider timeout.

Why: A transaction holds its locks until commit, so putting a network call inside one lends your database's concurrency to a third party's p99. Worse, a rollback cannot undo the external side effect, so the transaction's atomicity becomes a lie exactly when it matters — which is why the charge lives in the gap between two short transactions, with an idempotency key making the gap recoverable.

Checkout — three short transactions and one gap

Client

Idempotency-Key

Order API

TX 1 — reserve

stock decrement + order (pending_payment)

Payment provider

OUTSIDE any transaction

TX 2 — record

order → paid + outbox row

Outbox relay

reads committed rows only

Fulfilment

own transaction, idempotent

Reconciliation

daily: provider vs orders

The three transaction boundaries, and what is deliberately outside them

The three transaction boundaries, and what is deliberately outside them
Step	Inside one local transaction	Outside it	Why the split
1. Reserve	Decrement available stock · insert order (pending_payment) · insert reservation with an expiry	Nothing	Stock and order must agree or neither happens
2. Charge	—	The provider call, with an idempotency key and a timeout	A network call inside a transaction holds locks for someone else's latency
3. Record	Update order to paid or payment_failed · release stock on failure · insert outbox row	Nothing	The state change and the event it publishes must be one atomic write
4. Fulfil	Consumer's own local transaction	Reads the outbox asynchronously	Fulfilment failure must not roll back a completed payment

The gap between charge and record — both directions

The gap between charge and record — both directions
What happened	Symptom	Resolution
Charged, we timed out before recording	Money taken, order stuck in pending_payment	Re-send the same idempotency key: the provider returns the original charge, not a second one
Not charged, we recorded paid	Only possible if you record before the provider confirms — do not	Prevented by ordering, not by cleanup
Charged, then we crashed permanently	Order never leaves pending_payment	A reconciliation job compares provider charges to orders daily and flags mismatches
Provider webhook arrives before our own record	A payment event for an order still marked pending	The webhook handler must be idempotent and safe to arrive first

Remember: Three short local transactions with the provider call deliberately outside them: reserve stock and create the order, charge with an idempotency key, then record the result and write an outbox row atomically. Never put a network call inside a transaction — it lends your locks to a third party and its rollback cannot undo the charge. Let the database decide contention with a conditional update rather than read-then-write. Expect the gap between charged and recorded to happen in both directions, and reconcile against the provider rather than hoping retries cover it.

Stateful connections and shared infrastructure

The one project that cannot scale by adding stateless instances, and the one where every path must be isolated.

Project 5 — Chat platform

coreadvanced

Build real-time messaging: rooms, live delivery, history, and a sensible experience for someone who was offline. Its lesson is stateful connections. Every other project here scales by adding stateless instances behind a load balancer; this one cannot, because a WebSocket lives on one specific process, and the recipient of a message is almost never connected to the same process as the sender. That single fact produces most of the design. You need a registry mapping user to instance, or — simpler and more common — a pub/sub channel per room that every instance subscribes to, so a sender publishes once and whichever instances hold the recipients deliver. You need message ordering that does not depend on clocks, because two servers' clocks disagree and a message is not ordered by when it was typed: give each room a monotonic sequence assigned by the database on insert, and clients order by that. You need persistence before delivery, not after, because a message that was delivered live and never stored disappears on refresh. Offline sync then becomes simple — the client stores the last sequence it saw and asks for everything after it. Presence is deliberately approximate: it is high-churn, low-value data that belongs in a keyed store with a TTL, never in your primary database.

Think of it as

A radio station with a logbook. The broadcast reaches whoever is tuned in right now, and the logbook records everything regardless. Listeners who were away do not ask the broadcast to repeat itself — they read the log from where they left off. Conflating the two is what makes people try to make the broadcast reliable, which it can never be.

text

Build order — each step works before the next

  1  REST: POST /rooms/{id}/messages,
     GET /rooms/{id}/messages?after=<seq>
     No sockets yet. It already works.
  2  Persist with a per-room sequence.
  3  WebSocket: subscribe to a room, receive
     new messages. Persist THEN publish.
  4  Multiple instances: pub/sub per room.
  5  Reconnect: client sends last_seq,
     server replays the gap over REST.
  6  Presence: heartbeat -> keyed store,
     30s TTL. Absence means offline.
  7  Push when no live connection, deduped
     against the live delivery.

Definition of done: kill one instance while
a conversation is running; clients reconnect
and lose no message.

What we're doing: Handle a client that was disconnected for four minutes without losing or duplicating a message.

offline-sync.txttext

THE TEMPTING DESIGN

  "Buffer undelivered messages per user and
   replay them when they reconnect."

  Now the server tracks, per user:
    what was sent, what was acknowledged,
    what to retry, when to give up, and how
    much to buffer for someone offline for
    a week.
  All of that already exists, correctly, in
  the messages table. The buffer is a second
  copy of the truth with its own bugs.

THE DESIGN THAT WORKS

  The client remembers one number: the
  highest room sequence it has seen.

  Disconnect at seq 1041.
  Four minutes pass; 17 messages arrive.
  Reconnect:
    GET /rooms/7/messages?after=1041
    -> 17 messages, seq 1042..1058
    resubscribe to the live stream

  The server buffers nothing. The client's
  cursor is the only state, and the client
  is the one thing that always knows what it
  actually rendered.

THE RACE AT RECONNECT

  Between the REST fetch and the subscribe,
  a new message can slip through and be
  missed -- or arrive twice, if the order is
  reversed.
  Subscribe FIRST, buffer live messages in
  memory, then fetch history, then merge and
  drop anything with seq <= the last fetched.
  Duplicates are cheap to remove because the
  sequence makes them identifiable.
  A gap would not be.

WHY THE SEQUENCE, NOT A TIMESTAMP

  Two messages a millisecond apart on two
  servers can carry timestamps in the wrong
  order. Two readers then render the
  conversation differently, which is a
  correctness bug people report as "the
  replies are jumbled".
  A per-room sequence from the database has
  one writer and therefore one order.

THE COST, NAMED

  A room's sequence is a per-room write
  bottleneck. For human conversation, far
  below any real limit. For a 500k-viewer
  livestream chat it is not, and that design
  drops strict ordering deliberately.

12: Any per-user delivery buffer is a second copy of the message log with weaker guarantees. Recognising that early removes about half the state a naive chat design would carry.
26: Putting the cursor on the client is what makes the server stateless again apart from the socket itself — and the client is the only party that actually knows what it rendered.
36: Subscribe-then-fetch is the correct order because it converts a possible gap into a possible duplicate, and the sequence number makes duplicates trivially removable.
54: Naming the bottleneck keeps the choice honest: strict per-room ordering is affordable for conversation and not for a livestream, and the difference is a measured write rate.

Why this works: Offline sync is where a chat design either collapses into a small amount of durable state plus a cursor, or grows into per-user delivery tracking that never quite works. The deciding move is treating the live socket as an optimisation over the message log rather than as the delivery mechanism.

Fanning out before persisting

Wrong

text

bus.publish(room, message)     # instant
db.insert(message)             # <- fails
# Recipients saw the message live. It is not
# in history. On refresh it disappears, and
# the two people in the room now disagree
# about what was said -- with no error
# anywhere to explain it.

Better

text

row = db.insert(message)   # assigns seq
bus.publish(room, row)     # carries the seq
# If the insert fails the sender gets an
# error and nobody saw a phantom message.
# If the publish fails, the message is in
# history and appears on the next fetch.

What you see: Messages that were definitely on screen and are gone after a refresh, reported by users and unreproducible by engineers — because the failure lives in a write that no one is watching and that left no trace on the recipient's side.

Why: Publishing first makes the live stream the source of truth for a moment, and it is the one component with no durability at all. Persisting first also gives you the sequence number to publish, so ordering and durability are solved by the same ordering of two lines.

Chat — persist first, then fan out across instances

Sender

socket on instance A

Instance A

Messages

assigns room sequence

Pub/sub

one channel per room

Instance B

Recipient

socket on instance B

Presence store

heartbeat + 30s TTL

Push notification

only if no live socket

Two ways for instance A to reach a recipient connected to instance B

Two ways for instance A to reach a recipient connected to instance B
Approach	How it works	Cost	Choose when
Pub/sub per room	Every instance subscribes to the rooms its connections are in; sender publishes once	Every instance receives every message for a room it holds any member of	Almost always — it is simpler and has no registry to keep correct
Connection registry	A shared store maps user → instance; the sender routes directly	The registry must be kept accurate through crashes, and a stale entry silently drops messages	Very large rooms, where broadcasting to every instance is genuinely wasteful

Three sources of "order", and why only one works

Three sources of "order", and why only one works
Ordering by	Fails when	Verdict
Client timestamp	A device clock is wrong, or deliberately set wrong	Never — it is user-controlled input
Server receive time	Two servers' clocks differ by milliseconds, which is normal	No — produces different orders for different readers
Per-room sequence from the database	Only under a room-level write bottleneck, which is measurable	Yes — one writer per room defines one order for everyone

Remember: A WebSocket is pinned to one process, so cross-instance delivery is pub/sub per room. Persist before you fan out: the insert assigns a per-room sequence, and that sequence — never a clock — is what orders the conversation for every reader. Offline sync is a client-held cursor plus `?after=<seq>`, not a server-side delivery buffer; subscribe before fetching history so a gap becomes a removable duplicate. Presence goes in a keyed store with a TTL, where expiry is the offline signal.

Project 6 — Multi-tenant SaaS platform

coreadvanced

Build one system that serves many customer organisations from shared infrastructure. The lesson is that tenant isolation is a property, not a control: it has to hold on every path that touches data, and the paths people miss are the ones that do not go through a request handler. A query in a controller is easy to scope. A cache key is not — omit the tenant from it and one company sees another's data, with no error and no log line. A background job has no request context at all, so it has to carry a tenant id explicitly or it will happily run across everyone. Exports, search indexes, webhooks and log messages are the same story. The strongest structural answer is to make the tenant id impossible to forget rather than mandatory to remember: a database session variable with row-level security, a repository layer that will not build a query without one, cache keys constructed by a helper that requires it. RBAC sits inside the tenant boundary and is a different question — isolation says which rows exist for you at all, roles say what you may do with them, and conflating the two produces a system where an admin of one organisation can act on another. Quotas and billing then both need the same per-tenant usage counters, so build them once.

Think of it as

A shared office building. Isolation is the door lock on each suite; RBAC is who inside a suite may open the filing cabinet. The dangerous gaps are never the front doors — they are the shared cleaning contractor, the shared post room and the shared bins, which serve every floor and belong to no tenant.

text

Build order — each step works before the next

  1  tenants table; tenant_id on every row.
  2  Middleware resolves tenant from the
     token, sets a session variable.
  3  Row-level security policy per table
     using that variable. Now an unscoped
     query returns zero rows, not a leak.
  4  RBAC inside the tenant: roles ->
     permissions, checked per object.
  5  Cache keys via a builder requiring
     tenant id. No raw string keys.
  6  Background jobs carry tenant_id and set
     the session variable before querying.
  7  Usage counters per tenant -> quotas
     (enforce) and billing (invoice).

Definition of done: delete the WHERE clause
from one query and watch it return nothing
rather than another tenant's rows.

What we're doing: Find the leak that no test catches — a cache key without a tenant.

cache-key-leak.txttext

THE CODE

  def dashboard(user):
      key = f"dashboard:{user.id}"
      hit = cache.get(key)
      if hit: return hit
      data = repo.dashboard_for(user)   # scoped
      cache.set(key, data, ttl=300)
      return data

  The query is scoped. Row-level security is
  on. Every test passes.

THE LEAK

  User ids are unique per tenant, not
  globally -- they restart at 1 for each new
  organisation.
  Tenant A's user 1 loads the dashboard.
  Tenant B's user 1 loads the dashboard 10
  seconds later and gets A's cached data.

  No exception. No log line. No failing test,
  because every test runs with one tenant.
  The database was never asked.

WHY TESTS DO NOT CATCH IT

  A single-tenant test fixture makes the
  tenant id a constant, so omitting it from
  a key changes nothing. The bug is only
  reachable with two tenants whose ids
  collide -- which is the normal case in
  production and the rare case in a fixture.

THE STRUCTURAL FIX

  def cache_key(tenant_id, *parts):
      return ":".join([str(tenant_id), *parts])

  Ban raw string keys in review. A key that
  cannot be built without a tenant id cannot
  be built wrong.

  Then make the test suite multi-tenant by
  default: every fixture creates two tenants
  with colliding local ids, so this class of
  bug fails loudly the first time.

THE SAME BUG, THREE MORE PLACES

  - a background job that iterates users
    without a tenant filter
  - a search index shared across tenants
    with no mandatory filter clause
  - an error report that embeds the record
    that failed, sent to a shared tracker

15: Per-tenant sequential ids are what turn a missing prefix from a cache miss into a cross-tenant read. Global identifiers would make this leak impossible, which is one real argument for them.
23: The absence of any signal is what makes cache-key leaks the worst class of isolation bug: nothing fails, so the only discovery path is a customer noticing another company's data.
37: Making the tenant a required argument moves the guarantee from discipline to the type of the function. That distinction is the whole design principle of this project.
44: A multi-tenant test fixture with colliding local ids converts an invisible class of bug into a loud one. It costs a few lines in test setup and finds every future instance.

Why this works: Isolation bugs concentrate wherever data is keyed by something other than the database — caches, indexes, files, log payloads. The reliable fix is never "remember the tenant"; it is removing every API that lets you construct the artefact without one.

Running background jobs without a tenant context

Wrong

text

@nightly
def recalculate_usage():
    for account in Account.objects.all():
        ...
# No request, so no session variable, so no
# row-level security context. This iterates
# every tenant, and any per-account write it
# makes is unscoped too.

Better

text

@nightly
def enqueue_usage_jobs():
    for tid in active_tenant_ids():
        queue.publish(RecalcUsage(tenant=tid))

def handle(job):
    with tenant_context(job.tenant):   # sets
        for account in Account.objects.all():
            ...                        # scoped
# One job per tenant. Also isolates failure:
# one tenant's bad data no longer stops the
# other 400.

What you see: A nightly job that works for a year and then, after one tenant's data grows, times out for everybody — or worse, writes a value computed across all tenants into each tenant's record.

Why: Row-level security and scoped repositories both hang off a context that a request establishes and a scheduler does not. Fanning out to one job per tenant restores the context, and it converts a single shared failure domain into per-tenant ones, which is what you wanted for quotas and billing anyway.

Isolation has to hold on every path, not just the request

The path everyone scopes

HTTP request

tenant from token

Query

row-level security

The paths people forget

Cache key

no error when wrong

Background job

no request context

Export / report

Search index

Logs and traces

Webhook delivery

Inside the boundary

RBAC

what you may do

Quotas

from usage counters

Billing

same counters

Three isolation models, and what each actually buys

Three isolation models, and what each actually buys
Model	Shape	Isolation strength	Cost
Shared schema, tenant_id column	One database, one set of tables, a column on every row	Depends entirely on every query being scoped — enforce with row-level security	Cheapest to run; a single missing filter is a full breach
Schema per tenant	One database, one schema per tenant	A connection is scoped by search path; cross-tenant queries are unnatural	Migrations run per schema; hundreds of schemas get slow
Database per tenant	Separate database, sometimes separate instance	Strongest; also gives per-tenant backup and restore	Most expensive to operate; connection pools multiply

The paths where isolation is usually forgotten

The paths where isolation is usually forgotten
Path	The leak	The structural fix
Cache key	user:42:dashboard collides across tenants	A key builder that takes tenant id as a required argument
Background job	A nightly job iterates all rows with no tenant filter	Tenant id in the job payload; the worker sets the session variable before any query
Search index	One index for everyone; a query returns another tenant's documents	Tenant id as a mandatory filter clause, or an index per tenant
Export / report	A CSV built by a different code path than the UI	Exports go through the same scoped repository as everything else
Logs and errors	A stack trace or error payload containing another tenant's record	Log identifiers, not payloads; scrub before shipping
Webhook delivery	An event delivered to the wrong tenant's endpoint	Resolve the destination from the event's tenant, never from a shared config

Remember: Tenant isolation is a property that must hold on every path, and the leaks are on the paths with no request context: cache keys, background jobs, search indexes, exports, webhooks and logs. Enforce it structurally — row-level security, a repository that cannot build an unscoped query, a cache-key builder that requires a tenant — rather than by remembering. RBAC lives inside the boundary: isolation says which rows exist for you, roles say what you may do with them. Quotas and billing share one set of per-tenant usage counters.

Derived stores and atomic decisions

An index you can always rebuild, and a limit that holds across twelve servers.

Project 7 — Search platform

coreadvanced

Build search over a corpus you own: take documents in, index them, and serve ranked queries. The organising idea is that the index is derived, never authoritative. Every document has a home in a primary store, and the index is a projection of it that you must be able to throw away and rebuild at any moment. That single rule decides most of the design. Ingestion becomes a pipeline from the primary store to the index rather than a separate write path, so nothing can be searchable that does not exist. Failure recovery becomes "rebuild from the primary" rather than "restore an index backup", which is both simpler and always correct. And reindexing stops being an emergency: you will change the analyzer, add a field or fix a mapping far more often than you expect, and each of those needs a full rebuild. The way to make that routine is aliases — build the new index alongside the live one, then atomically swing a name from old to new, keeping the old one until you are sure. The other thing this project teaches is that ranking is a product decision with an evaluation loop, not a configuration value: you need a set of queries with known good results before you can tell whether a ranking change helped.

Think of it as

A library catalogue. The books are the truth; the catalogue is a convenience built from them, and a burnt catalogue means a slow week of re-cataloguing rather than lost books. Nobody would ever record a new acquisition only in the catalogue — and that is exactly the mistake a search design makes when it lets something be indexed without existing in the store.

text

Build order — each step works before the next

  1  Primary store + a naive query API using
     SQL LIKE. Slow, correct, a baseline.
  2  Index one document type. Full rebuild
     script from the primary, run by hand.
  3  Query API against the index. Compare
     results with the baseline.
  4  Ingestion: outbox/change stream ->
     indexer, with a durable cursor.
  5  Alias: search_v1 behind alias "search".
     Rebuild into v2, swing the alias.
  6  Evaluation set: 50 queries with known
     good results. Score before/after.
  7  Cache the hot query results.
  8  Degraded mode when the index is down.

Definition of done: delete the entire index
and have it back, correct, without touching
a backup.

What we're doing: Change the analyzer on a live index without a maintenance window or a wrong-results period.

reindex-with-alias.txttext

THE CHANGE

  Add stemming so "running shoes" matches
  "run shoe". That means a new analyzer,
  which means every existing document must
  be re-analyzed. There is no in-place
  version of this change.

WHAT PEOPLE TRY FIRST

  Update the mapping, reindex in place.
  For the length of the rebuild:
    - some documents are analyzed old-style,
      some new-style
    - results are a mixture, and neither
      correct nor obviously broken
    - if it fails halfway, you now have a
      permanently mixed index and no clean
      way back

WHAT THE ALIAS DOES

  Live:      alias "search" -> products_v1
  Build:     create products_v2 with the new
             analyzer; bulk-index from the
             primary store; let ingestion
             write to BOTH v1 and v2 while
             the rebuild runs
  Verify:    query v2 directly with the
             evaluation set; compare scores
  Swap:      alias "search" -> products_v2
             (atomic; one operation)
  Keep:      leave v1 in place for a day
  Rollback:  swing the alias back. Seconds.

  No maintenance window, no mixed state, and
  a rollback that is one command instead of
  another rebuild.

WHY DUAL-WRITE DURING THE REBUILD

  A rebuild that takes 40 minutes would
  otherwise miss 40 minutes of updates, so
  v2 would be stale at the moment it went
  live. Writing to both during the window
  keeps v2 current.
  Alternative: record the change-stream
  cursor before the rebuild starts, then
  catch up from it before the swap. Either
  works; the second needs less code and one
  more step.

HOW YOU KNOW IT WORKED

  Without an evaluation set, "did stemming
  help?" is a matter of opinion. With 50
  queries whose good results are known:
    v1: 34/50 with a good result in the
        top 3
    v2: 41/50
  That is a decision. Anything less is
  someone trying three searches they
  happen to remember.

9: Reindexing in place is the default action most search clients make easy, and it has no correct intermediate state — which is precisely why the alias indirection exists.
27: The alias swap is one atomic operation on the search cluster, so no query ever sees a half-built index. That is what turns a risky migration into a routine one.
41: Missing the write window is the mistake that makes a technically successful reindex ship stale data. Both fixes are cheap; not having one is what costs.
55: An evaluation set is the difference between a ranking change and a ranking guess. Fifty labelled queries is enough to catch a regression and takes an afternoon to build.

Why this works: Reindexing is not an exceptional operation — analyzers, mappings and fields change constantly — so the design either makes it routine or makes every schema change a small crisis. Aliases plus a rebuild-from-primary guarantee turn both reindexing and disaster recovery into the same well-practised procedure.

Writing to the index and the primary store separately

Wrong

text

db.insert(product)
search.index(product)     # <- fails, or
                          #    succeeds alone
# Two systems, two writes, no atomicity.
# Either a product exists and is unfindable,
# or -- worse -- search returns a product
# that was rolled back and does not exist.

Better

text

BEGIN;
  INSERT INTO products ...;
  INSERT INTO outbox (topic, payload);
COMMIT;
# The indexer consumes the outbox. Search
# lags by seconds and can never contain a
# document the store does not have. The lag
# is visible; the inconsistency is not
# possible.

What you see: Search results that 404 when clicked, and products that exist and cannot be found — two symptoms of one cause, usually reported as two unrelated bugs by two different teams.

Why: Dual writes have no ordering that survives a crash: index-first can invent documents, store-first can lose them, and neither can be made atomic across two systems. Deriving the index from a committed log in the primary store makes the index a function of the store, which is what "derived" actually requires.

Search — a derived index, rebuilt behind an alias

Primary store

the truth

Change stream

outbox or CDC · durable cursor

Indexer

analyze · map · write

Index v2

built alongside the live one

Alias "search"

atomic swap v1 → v2

Query API

parse · filter · rank

Result cache

hot queries, page 1

User

The pipeline, and the failure each stage owns

The pipeline, and the failure each stage owns
Stage	Does	Fails as	Recovery
Ingestion	Reads changes from the primary (outbox or change stream)	Lag — new documents are not yet searchable	Catch-up; the cursor is durable so nothing is lost
Indexing	Analyzes and writes documents into the index	Rejected documents, mapping conflicts	Dead-letter the document, alert, fix the mapping, reindex
Query API	Parses the query, applies filters, calls the index	Slow or failing index	Degrade to a primary-store filter, or return an honest error
Ranking	Orders results by relevance and business rules	Silently worse results	Only detectable against an evaluation set — nothing else will tell you
Reindex	Rebuilds the whole index under a new name	Half-built index serving traffic	Prevented by the alias swap; never index into the live one

What to cache, and what not to

What to cache, and what not to
Candidate	Cache it?	Why
Full query result page 1	Yes	Query distribution is heavily skewed; a small set of queries is most of the traffic
Deep pagination (page 40)	No	Almost never repeated, and expensive to hold
Individual documents	Rarely	The index already serves them fast; the win is in avoiding the query, not the fetch
Autocomplete prefixes	Yes	Extremely repetitive and latency-sensitive
Personalised results	Only per user, briefly	A shared cache key here is a cross-user leak, not a performance win

Remember: The index is derived, never authoritative — recovery is a rebuild from the primary store, not a restore. Ingest from a committed change stream or outbox so search can never contain a document the store does not have. Reindex behind an alias: build the new index alongside, catch up or dual-write, verify against an evaluation set, then swap atomically and keep the old one for a day. Cache query results rather than documents, and show index lag to the user instead of letting it look like an empty result.

Project 8 — Distributed rate limiter

coreintermediate

Build something that lets a caller through 100 times a minute and refuses the 101st, correctly, when the check is running on twelve application servers at once. It is the smallest project here and it teaches atomicity better than any of the others, because the obvious implementation — read the counter, compare it to the limit, write the new value — is wrong in a way that only shows up under concurrency, which is precisely the condition a rate limiter exists for. Under load, many servers read the same value before any of them writes, and the limit is exceeded by roughly the number of concurrent servers. The fix is to make the read, the decision and the write one operation the store performs indivisibly: an atomic increment, or a small script the store runs to completion without interleaving. Choosing the algorithm is a separate decision. A fixed window is trivial and allows twice the limit across a window boundary. A sliding window fixes that at the cost of more state. A token bucket allows a controlled burst and then a steady rate, which is usually what an API actually wants. The last decision is the one people skip: what happens when the store holding the counters is down — fail open and lose all protection, or fail closed and take an outage.

Think of it as

A nightclub with one doorman and twelve doors. Twelve doormen each counting to a hundred lets in twelve hundred people. The fix is not better counting — it is one shared counter that only one person can touch at a time, and that indivisibility is what the whole design is buying.

text

Build order — each step works before the next

  1  In-process fixed window. One server.
     Correct, and useless with two servers.
  2  Move the counter to Redis with INCR +
     EXPIRE. Now shared, still boundary-spiky.
  3  Make it atomic in ONE round trip
     (script or pipelined INCR+EXPIRE) --
     then prove it with concurrent load.
  4  Token bucket via a script: refill by
     elapsed time, take one, return allowed.
  5  Headers: X-RateLimit-Limit/Remaining/
     Reset, and Retry-After on 429.
  6  Decide and implement fail-open or
     fail-closed, with a short timeout.
  7  Per-plan quotas reading the same
     counters.

Definition of done: 12 concurrent clients, a
limit of 100, and exactly 100 get through.

What we're doing: Write the token bucket as one atomic operation, and see why it cannot be three.

token-bucket.txttext

THE ALGORITHM

  Each key holds two values:
    tokens        how many are left
    last_refill   when we last topped up

  On a request:
    elapsed = now - last_refill
    tokens  = min(capacity,
                  tokens + elapsed * rate)
    if tokens >= 1:
        tokens -= 1;  allow
    else:
        reject, retry after (1 - tokens)/rate

WHY IT MUST BE ONE OPERATION

  Written as read -> compute -> write, two
  servers both read tokens = 1, both compute
  0, both allow, and both write 0.
  The bucket allowed two requests while
  holding one token, and no individual step
  was wrong.

  Run as a script inside the store, the whole
  sequence executes without interleaving.
  Every caller sees a bucket that has already
  accounted for everyone before it.

WHAT THE SCRIPT RETURNS

  allowed (0/1), tokens remaining, and
  seconds until the next token. The last one
  becomes Retry-After, which is what turns a
  429 into something a client can obey rather
  than something it retries immediately.

WHY TOKEN BUCKET AND NOT FIXED WINDOW

  A client that makes 20 calls when a page
  loads and then nothing for a minute is
  normal, not abusive. A fixed window either
  rejects the page load or has to be set so
  high it stops protecting anything.
  A bucket of 20 refilling at 1/sec allows
  exactly that shape and still caps the
  sustained rate at 60/min.

THE FAILURE DECISION

  Redis is down. Two honest options:
    fail open  -> allow everything. The API
                  is unprotected but up.
    fail closed-> reject everything. Fully
                  protected, and fully down.
  For a public API protecting a fragile
  backend, fail closed on the write paths
  and open on cheap reads is a common split.
  What is not acceptable is having no
  timeout: then the limiter blocks on a dead
  store and every request hangs, which is
  fail-closed with extra latency.

18: This is the same shape as the last-unit inventory race in project 4. Recognising it as one pattern — a decision based on a value that can change before the write — is most of what these two projects teach.
27: A store-side script gives atomicity across several keys and some arithmetic, which a single atomic increment cannot. That is the reason token bucket needs one and a fixed window does not.
33: Returning the retry delay from the same operation that made the decision is what makes Retry-After accurate. Computing it separately reintroduces a small race for no benefit.
55: The no-timeout case is the one that actually happens: nobody chooses it, and it is strictly worse than either deliberate option because it also consumes a connection for the duration.

Why this works: The interesting content of a rate limiter is not the algorithm — all four are a few lines. It is that correctness depends entirely on where the decision happens, and moving the decision into the store is the difference between a limit that holds at twelve servers and one that holds at one.

Rate limiting by IP address alone

Wrong

text

key = f"rl:{request.remote_addr}"
# One office behind NAT shares an IP, so 200
# colleagues get 100 requests between them.
# An attacker with 500 IPs gets 50,000.
# The limit punishes the wrong people in
# both directions at once.

Better

text

# Authenticated traffic: key on the
# identity you issued.
key = f"rl:{api_key}:{endpoint_class}"
# Unauthenticated traffic: IP is what you
# have -- so use it with a much looser
# limit, and put the strict limits behind
# authentication.

What you see: Support tickets from your largest corporate customers about being rate limited, arriving in the same week that an abusive scraper is comfortably under the limit from a residential proxy pool.

Why: The identity key defines who shares a budget, so choosing it by what is easiest to read from a request rather than by who should be accountable groups unrelated callers together and splits a single caller apart. IP is a network fact, not an identity, and the two only coincide for a single user on a single connection.

Why GET-then-SET overshoots, and what atomic does

Server A

Server B

Redis

1. GET count:user42Naive version — three separate operations
2. → 99
3. GET count:user42Arrives before A has written anything back
4. → 99
5. SET 100 · allowA believes it took the last slot
6. SET 100 · allow101 requests allowed with a limit of 100 — and with 12 servers, ~112
7. INCR count:user42Atomic version — one operation, the store orders it
8. → 100 · allow
9. INCR count:user42
10. → 101 · rejectNo interleaving is possible: the read and the write are the same operation

Three algorithms, with the burst each one permits

Three algorithms, with the burst each one permits
Algorithm	State per key	Burst behaviour	Pick it when
Fixed window	One counter + expiry	Up to 2× the limit across a boundary (100 at 11:59:59, 100 at 12:00:00)	Rough protection is enough and simplicity matters most
Sliding window log	A timestamp per request	Exact — no boundary effect at all	Limits must be precise and request volume per key is modest
Sliding window counter	Two counters, weighted by position in the window	Approximate, no boundary spike, small fixed state	A good default: nearly exact, cheap
Token bucket	Token count + last refill time	A deliberate burst up to bucket size, then a steady rate	Public APIs — it matches how clients actually behave

The decisions that are not the algorithm

The decisions that are not the algorithm
Decision	Options	Consequence of choosing badly
Identity key	user id · API key · IP · tenant · endpoint + user	IP alone throttles an entire office behind one NAT and misses a distributed attacker
Scope	Global · per endpoint · per tenant · per plan	One global limit means an expensive endpoint and a cheap one share a budget
Store failure	Fail open (allow) · fail closed (reject)	Undecided means fail closed by timeout — a counter outage becomes a full outage
Response	429 + Retry-After + limit headers	Without Retry-After, well-behaved clients retry immediately and make it worse
Enforcement point	Edge/gateway · application · both	Application-only means rejected traffic still costs you a connection and a process

Remember: Read-compare-write is wrong under concurrency; make the check and the update one atomic store operation — an increment, or a script the store runs without interleaving. Pick the algorithm by the burst you want to allow: fixed window is spiky at boundaries, sliding window counter is a cheap near-exact default, token bucket allows a bounded burst and matches real API clients. Then answer the three questions that are not the algorithm: what identity shares a budget, what happens when the counter store is down, and whether the flood is being rejected at the edge or deep inside your application.