Eight projects that validate the theory

Build a service that turns a long URL into a short code and redirects that code back to the original. It looks small, and that is why it is the first project: the whole thing fits in a weekend, but it forces four decisions that recur everywhere. The first is the read/write asymmetry — redirects outnumber creations by a factor of hundreds or thousands, so the create path and the redirect path get designed separately rather than as two endpoints on the same service. The second is ID generation, which has no obviously correct answer until you name the property you want: a random code is unguessable but needs a collision check, a counter in base62 is short and dense but leaks how many links exist and needs coordination between instances, and a hash of the URL deduplicates identical links but makes two users share one code and therefore one set of analytics. The third is caching, because the redirect path is a pure key lookup and belongs in Redis — with the database still the system of record, so a cache flush costs latency rather than data. The fourth is that analytics must not be on the redirect path: a click writes an event to a queue and the redirect returns, because a redirect that waits on an analytics write has coupled its availability to a system nobody would page for.

Build order — each step works before the next

  1  POST /links + GET /{code}, Postgres only
  2  ID generation: pick a scheme, write down
     which property you chose it for
  3  Redis read-through on the redirect path
  4  Click events -> queue -> analytics store
  5  Rate limit on POST /links (per API key)
  6  Expiry: TTL column + a sweep, 410 on
     an expired code

Definition of done: you can state your p99
redirect latency, what happens when Redis is
down, and why you chose 301 or 302.

Build a service other services call to send email, SMS and push. It teaches the intent/delivery split better than anything else, because the split is unavoidable here: one accepted request ("tell user 91 their order shipped") fans out into several attempts across several channels, each with its own state, its own provider and its own failure. So there are two tables, not one. A notification row records what was asked for and is created inside the caller's request. Delivery rows record each attempt on each channel and are created by workers afterwards. Separate worker pools per channel matter more than they look: email providers rate-limit differently from SMS providers, and one pool means a backlog of SMS retries starves email. Deduplication is the other unavoidable piece — callers retry, and a retried "order shipped" must not send a second email, so the caller supplies a key and the same key returns the same notification. Templates belong on the platform rather than in each caller so that a wording fix is one deploy. And delivery tracking is what makes the whole thing operable: without a per-attempt state you cannot answer "did user 91 get it", which is the only question anyone ever asks.

Build order — each step works before the next

  1  POST /notifications -> row + 202 + id
     GET /notifications/{id} -> status
  2  Queue + one email worker. Provider call,
     store the provider message id.
  3  Delivery rows, one per channel attempt.
  4  Retries: 5 attempts, backoff + jitter,
     then dead-letter. 429 slows the pool.
  5  Dedupe key on POST, unique per caller.
  6  Templates, versioned, rendered by the
     worker so a fix is one deploy.
  7  SMS + push as separate pools.
  8  Provider webhooks -> delivery status.

Definition of done: you can answer "did user
91 get the shipping email, and if not, why"
from stored state alone.

Build something that accepts a large file, does slow work on it, and gives the result back — resizing images, transcoding video, extracting text from a PDF. The central lesson is that the bytes never touch your application servers. The client asks the API for a signed upload URL, uploads directly to object storage, then tells the API the upload finished; the API stores metadata and enqueues a job. Doing it the other way, streaming the file through your API, ties every upload to one server's memory, bandwidth and deploy cycle, and a rolling restart kills every upload in flight. The second lesson is that progress tracking is a requirement, not a nicety: work measured in minutes needs a job record with a state, a percentage and an error field, because a client that cannot poll will either retry (doubling the work) or hang. The third is the multipart upload — a 4 GB file over a phone connection will fail partway, and multipart lets it resume from the failed part rather than from zero. And the whole thing has an ownership rule worth stating: object storage holds bytes, the database holds truth. A file that exists in storage with no row is garbage to be swept; a row with no object is a failed upload to be reported.

Build order — each step works before the next

  1  POST /jobs -> job row + signed PUT URL
     Client PUTs the object directly.
  2  POST /jobs/{id}/complete -> validate the
     object exists, size + type, then enqueue
  3  Worker: claim with a lease, process,
     write output object, update job
  4  GET /jobs/{id} -> state + percent
  5  Signed GET URL for the result, short TTL
  6  Multipart upload for files over ~100 MB
  7  Sweep: objects with no row, rows stuck
     in awaiting_upload past their URL expiry

Definition of done: kill a worker mid-job and
the job completes anyway, exactly once.

Build checkout: reserve stock, take payment, create an order, then fulfil it. This is the hardest of the eight projects and the one worth the most, because it is the smallest realistic system where money, inventory and a third party all have to agree. Three things make it hard. First, the payment provider is outside your database, so no transaction can cover both the charge and the order row — the design has to decide what happens in the gap between them, in both directions. Second, inventory is contended: two customers want the last unit at the same moment, and the database, not the application, has to be the thing that decides. Third, every step is retried by somebody — the customer refreshes, the client times out and retries, the queue redelivers — so every write needs to be safe to run twice. The shape that handles all three is: one short local transaction that reserves stock and creates the order in `pending_payment`, then the payment attempt outside any transaction, then a second short transaction that records the result and writes an outbox row, then fulfilment consuming that outbox asynchronously. The rule underneath it is that a database transaction must never contain a network call to something you do not control.

Build order — each step works before the next

  1  Orders + items. POST /orders with an
     Idempotency-Key; same key -> same order.
  2  Inventory: conditional decrement
       UPDATE stock SET qty = qty - 1
       WHERE sku = ? AND qty >= 1
     0 rows updated = out of stock. No read
     first. The database decides.
  3  Payment outside any transaction, with
     the provider's idempotency key = your
     payment attempt id.
  4  Outbox row written in the same
     transaction as the order state change.
  5  Relay publishes outbox -> fulfilment
     consumes, idempotently.
  6  Reservation expiry sweep: unpaid holds
     return stock after N minutes.
  7  Reconciliation job vs the provider.

Definition of done: kill the process between
every pair of steps; no money is lost and no
stock is double-sold.

Build real-time messaging: rooms, live delivery, history, and a sensible experience for someone who was offline. Its lesson is stateful connections. Every other project here scales by adding stateless instances behind a load balancer; this one cannot, because a WebSocket lives on one specific process, and the recipient of a message is almost never connected to the same process as the sender. That single fact produces most of the design. You need a registry mapping user to instance, or — simpler and more common — a pub/sub channel per room that every instance subscribes to, so a sender publishes once and whichever instances hold the recipients deliver. You need message ordering that does not depend on clocks, because two servers' clocks disagree and a message is not ordered by when it was typed: give each room a monotonic sequence assigned by the database on insert, and clients order by that. You need persistence before delivery, not after, because a message that was delivered live and never stored disappears on refresh. Offline sync then becomes simple — the client stores the last sequence it saw and asks for everything after it. Presence is deliberately approximate: it is high-churn, low-value data that belongs in a keyed store with a TTL, never in your primary database.

Build order — each step works before the next

  1  REST: POST /rooms/{id}/messages,
     GET /rooms/{id}/messages?after=<seq>
     No sockets yet. It already works.
  2  Persist with a per-room sequence.
  3  WebSocket: subscribe to a room, receive
     new messages. Persist THEN publish.
  4  Multiple instances: pub/sub per room.
  5  Reconnect: client sends last_seq,
     server replays the gap over REST.
  6  Presence: heartbeat -> keyed store,
     30s TTL. Absence means offline.
  7  Push when no live connection, deduped
     against the live delivery.

Definition of done: kill one instance while
a conversation is running; clients reconnect
and lose no message.

Build one system that serves many customer organisations from shared infrastructure. The lesson is that tenant isolation is a property, not a control: it has to hold on every path that touches data, and the paths people miss are the ones that do not go through a request handler. A query in a controller is easy to scope. A cache key is not — omit the tenant from it and one company sees another's data, with no error and no log line. A background job has no request context at all, so it has to carry a tenant id explicitly or it will happily run across everyone. Exports, search indexes, webhooks and log messages are the same story. The strongest structural answer is to make the tenant id impossible to forget rather than mandatory to remember: a database session variable with row-level security, a repository layer that will not build a query without one, cache keys constructed by a helper that requires it. RBAC sits inside the tenant boundary and is a different question — isolation says which rows exist for you at all, roles say what you may do with them, and conflating the two produces a system where an admin of one organisation can act on another. Quotas and billing then both need the same per-tenant usage counters, so build them once.

Build order — each step works before the next

  1  tenants table; tenant_id on every row.
  2  Middleware resolves tenant from the
     token, sets a session variable.
  3  Row-level security policy per table
     using that variable. Now an unscoped
     query returns zero rows, not a leak.
  4  RBAC inside the tenant: roles ->
     permissions, checked per object.
  5  Cache keys via a builder requiring
     tenant id. No raw string keys.
  6  Background jobs carry tenant_id and set
     the session variable before querying.
  7  Usage counters per tenant -> quotas
     (enforce) and billing (invoice).

Definition of done: delete the WHERE clause
from one query and watch it return nothing
rather than another tenant's rows.

Build search over a corpus you own: take documents in, index them, and serve ranked queries. The organising idea is that the index is derived, never authoritative. Every document has a home in a primary store, and the index is a projection of it that you must be able to throw away and rebuild at any moment. That single rule decides most of the design. Ingestion becomes a pipeline from the primary store to the index rather than a separate write path, so nothing can be searchable that does not exist. Failure recovery becomes "rebuild from the primary" rather than "restore an index backup", which is both simpler and always correct. And reindexing stops being an emergency: you will change the analyzer, add a field or fix a mapping far more often than you expect, and each of those needs a full rebuild. The way to make that routine is aliases — build the new index alongside the live one, then atomically swing a name from old to new, keeping the old one until you are sure. The other thing this project teaches is that ranking is a product decision with an evaluation loop, not a configuration value: you need a set of queries with known good results before you can tell whether a ranking change helped.

Build order — each step works before the next

  1  Primary store + a naive query API using
     SQL LIKE. Slow, correct, a baseline.
  2  Index one document type. Full rebuild
     script from the primary, run by hand.
  3  Query API against the index. Compare
     results with the baseline.
  4  Ingestion: outbox/change stream ->
     indexer, with a durable cursor.
  5  Alias: search_v1 behind alias "search".
     Rebuild into v2, swing the alias.
  6  Evaluation set: 50 queries with known
     good results. Score before/after.
  7  Cache the hot query results.
  8  Degraded mode when the index is down.

Definition of done: delete the entire index
and have it back, correct, without touching
a backup.

Build something that lets a caller through 100 times a minute and refuses the 101st, correctly, when the check is running on twelve application servers at once. It is the smallest project here and it teaches atomicity better than any of the others, because the obvious implementation — read the counter, compare it to the limit, write the new value — is wrong in a way that only shows up under concurrency, which is precisely the condition a rate limiter exists for. Under load, many servers read the same value before any of them writes, and the limit is exceeded by roughly the number of concurrent servers. The fix is to make the read, the decision and the write one operation the store performs indivisibly: an atomic increment, or a small script the store runs to completion without interleaving. Choosing the algorithm is a separate decision. A fixed window is trivial and allows twice the limit across a window boundary. A sliding window fixes that at the cost of more state. A token bucket allows a controlled burst and then a steady rate, which is usually what an API actually wants. The last decision is the one people skip: what happens when the store holding the counters is down — fail open and lose all protection, or fail closed and take an outage.

Build order — each step works before the next

  1  In-process fixed window. One server.
     Correct, and useless with two servers.
  2  Move the counter to Redis with INCR +
     EXPIRE. Now shared, still boundary-spiky.
  3  Make it atomic in ONE round trip
     (script or pipelined INCR+EXPIRE) --
     then prove it with concurrent load.
  4  Token bucket via a script: refill by
     elapsed time, take one, return allowed.
  5  Headers: X-RateLimit-Limit/Remaining/
     Reset, and Retry-After on 429.
  6  Decide and implement fail-open or
     fail-closed, with a short timeout.
  7  Per-plan quotas reading the same
     counters.

Definition of done: 12 concurrent clients, a
limit of 100, and exactly 100 get through.