beginnerintermediateadvanced

System Design · Section 83

Notification System Design

Level: intermediate
Read: 14 min
Concepts: 2

A notification has two halves that belong apart. The intent is the business fact — "this recipient should learn that their order shipped" — which is channel-independent, produced by the service that owns the event, and recorded once, ideally in the same transaction as the business change so that "shipped but nobody was told" is a state the database refuses to produce. The delivery is one attempt to reach a person over one channel, with its own address, its own provider, its own retries and its own terminal failures. Keeping them separate stops an email provider outage from becoming an order-processing outage, and keeps channel, template and preference knowledge in the one service that owns it rather than in every service that ever notifies anyone. The delivery half then needs five things. Queueing, so a slow provider becomes queue depth rather than blocked requests and send rate becomes something you control. Per-attempt delivery state, keeping `sent` (the provider accepted the request) distinct from `delivered` (a webhook confirmed arrival), because a message can be accepted and bounce minutes later and only the second event distinguishes a working address from a dead one. Retry handling that separates transient failures — 429, 503, timeouts — from permanent ones like a hard bounce, which must never be retried because repetition damages sender reputation. Deduplication on a key derived from the event rather than the attempt, so an at-least-once queue redelivery is recognised rather than sent twice. And two independent limits resolved at send time rather than at intent time: recipient preferences, which decide whether a channel may be used at all and can change while a message sits queued, and provider rate limits, which decide how fast it may be used.

This section

System Design Interview - Notification ServiceSystem Design Interview

What is true here

Intent is a channel-independent fact written in the business transaction; delivery is one channel attempt with its own retries and failure states.
A synchronous send inside a business transaction makes that transaction inherit the provider's availability — an email outage becomes an order outage.
sent means the provider accepted the message; delivered needs a webhook. Collapsing them hides every bounce and lets bad addresses degrade sender reputation.
Retry transient failures with backoff; never retry a hard bounce — suppress the address instead.
Resolve preferences and provider limits at send time, because a queued message can wait long enough for an opt-out to arrive in between.

What you will be able to do

Split a notification flow into an intent record and per-channel delivery records with distinct lifecycles
Write the intent in the same transaction as the business change so no event goes silently unannounced
Design a delivery state machine that distinguishes provider acceptance from confirmed delivery and from suppression
Choose a deduplication key that survives at-least-once redelivery, and decide which failures are worth retrying

Intent versus delivery

The business fact and the channel attempt as two records with two owners and two lifecycles.

Separating notification intent from delivery

coreintermediate

A notification has two halves that want to live apart. The intent is the business fact: "order 8814 shipped, and this customer should be told." It is produced by the code that knows the business event, it is true regardless of any channel, and it should be recorded once. The delivery is one attempt to get that fact to a person over one channel: this email to this address through this provider, that push to that device token. One intent can produce several deliveries — email plus push — and each delivery can fail, retry and succeed independently, or be suppressed entirely because the person turned that channel off. Collapsing the two halves means the code that ships an order also formats HTML, looks up an SMTP provider and handles a bounce, which couples business logic to an unrelated failure domain: the shipping transaction now fails when the email provider is slow. Splitting them gives the order service one job — record the intent — and gives a separate notification service every job after that. It also makes the two questions people actually ask answerable separately: "did we decide to tell this customer" and "did the message reach them" are different questions with different answers, and a design with one record cannot distinguish them.

Think of it as

A newsroom decides a story is worth publishing; the press decides how many copies go out and where. The decision is recorded once and stays true even if the delivery van breaks down; the van breaking down is a delivery problem, not a reason the story stops being news. Your order service is the newsroom — it decides something is worth telling someone — and your notification service is the press, the vans and the routes. Confusing the two means a broken van retracts the story.

sql

-- the fact, written in the business transaction
CREATE TABLE notification_intents (
  id           uuid PRIMARY KEY,
  event_type   text NOT NULL,      -- 'order.shipped'
  recipient_id uuid NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL
);

-- the attempts, owned by the notification service
CREATE TABLE notification_deliveries (
  id        uuid PRIMARY KEY,
  intent_id uuid NOT NULL REFERENCES notification_intents,
  channel   text NOT NULL,         -- 'email' | 'push'
  state     text NOT NULL,         -- queued|sent|bounced
  attempts  int  NOT NULL DEFAULT 0
);

What we're doing: Compare a coupled design and a split design when the email provider has a 40-minute outage.

email-outage.txttext

Coupled: ship_order() sends the email inline

  09:00  ship_order(8814)
         -> smtp.send(...) hangs 30s, times out
         -> the whole ship_order transaction
            rolls back
  Result: the order is NOT shipped. A mail
  outage became an order-processing outage.
  During 40 minutes, 900 orders fail to ship.

Split: ship_order() records an intent

  09:00  ship_order(8814) commits, with an
         intent row in the same transaction
  09:00  notification worker picks it up,
         email delivery attempt 1 fails
  09:02  attempt 2 fails (backoff)
  09:41  attempt 6 succeeds
  Result: 900 orders shipped on time. 900
  emails arrive up to 41 minutes late, which
  is the actual severity of a mail outage.

5: This is the whole argument in one line: a synchronous side effect inside a business transaction makes the transaction inherit that side effect's availability. The order database was healthy the entire time.
13: Writing the intent in the same transaction as the business change is what makes the split safe. If the intent were written after the commit, a crash in between would ship an order that nobody is ever told about — see the Transactional Outbox pattern for the full mechanism.
20: The degraded outcome is now proportionate: late emails during a mail outage. Nothing about the outage touched order processing, because nothing about order processing depended on mail.

Why this works: The split does not make the email provider more reliable — it changes which part of your system is exposed to that provider's reliability. Coupled, an email outage is an order outage; split, an email outage is late email. The failure that matters is the one that reaches a customer, and only one of these two designs keeps it small.

Writing the intent outside the business transaction

Wrong

python

with db.transaction():
    order.state = "shipped"
# commit happens here
notify.record_intent(order)   # a crash here loses
                              # the notification
                              # forever, silently

Better

python

with db.transaction():
    order.state = "shipped"
    db.insert("notification_intents", {...})
# both land or neither does; a relay picks the
# intent up afterwards and drives delivery

What you see: A small, steady fraction of customers never receive a notification for an event that definitely happened, with no failed delivery row to explain it — because no delivery was ever attempted, and no intent was ever recorded.

Why: Two writes to two places cannot both be guaranteed unless they are in one transaction. Recording the intent alongside the business change makes "shipped but nobody was told" a state the database will not produce, which is a stronger guarantee than any amount of retrying around a call that was never made.

One intent, several independent deliveries

Order shipped

business transaction commits

Record intent

one row, channel-independent

Resolve preferences

which channels is this recipient open to?

Email delivery

own retries, own failure states

Push delivery

own retries, own failure states

SMS delivery

suppressed — recipient opted out

Intent and delivery: two records, two owners, two lifecycles

Intent and delivery: two records, two owners, two lifecycles
Property	Intent	Delivery
Produced by	The service that owns the business event	The notification service
Cardinality	One per business event per recipient	Zero or more per intent — one per channel attempt
Content	Event type, recipient id, payload data	Channel, address, provider, rendered body
States	created → dispatched	queued → sent → delivered / bounced / suppressed
Retried?	No — it is a fact, not an attempt	Yes, with backoff, per channel
Survives a channel outage	Yes — the fact is already recorded	No — that is what a retry is for

Remember: Intent is the business fact — "this recipient should learn this happened" — recorded once, in the same transaction as the business change, with no channel in it. Delivery is one attempt over one channel, with its own retries, its own failure states and its own suppression rules. Keeping them apart stops a mail provider outage from becoming an order outage, and keeps channel and preference knowledge in the one service that owns it.

Running the delivery side

Queueing, delivery state, retry classification, deduplication, and the two limits resolved at send time.

Queueing, delivery state, retries, deduplication and preferences

coreintermediate

Once intent and delivery are separate, the delivery side needs five things. Queue the work, so a slow or failing provider adds queue depth instead of blocking a request — and so the number of messages you send per second is something you control rather than something traffic decides. Track delivery state per attempt (queued, sent, delivered, bounced, suppressed), because "we called the API and it returned 200" is not the same as "it reached a person": an email is accepted by a provider and can still bounce minutes later, and only a delivery-event webhook tells you which. Retry transient failures with backoff, and distinguish them from permanent ones — a 429 or a 503 is worth retrying, a hard bounce from a nonexistent address never is, and retrying it repeatedly damages your sender reputation. Deduplicate on a key derived from the event, not from the attempt, because at-least-once queues redeliver and a resumed worker will otherwise send the same message twice. And respect two independent limits: the recipient's preferences, which decide whether a channel may be used at all, and the provider's rate limits and quotas, which decide how fast you may use it — checked at send time rather than at intent time, so a preference changed between the two is honoured.

Think of it as

Picture a mail room with an in-tray, a ledger and a set of standing instructions. The in-tray is the queue: work piles up when the post office is slow, and nobody in the building has to wait. The ledger records what happened to each item — handed over, delivered, returned to sender, never sent because this recipient asked not to be written to. The standing instructions are the two limits: some recipients have said "no marketing post", and the post office will only take 500 items an hour. A mail room without a ledger cannot tell you whether a letter arrived; one without standing instructions eventually gets its account suspended.

python

def deliver(intent, channel):
    key = f"{intent.id}:{channel}"        # per event,
    if not claim(key):                    # not per
        return                            # attempt
    if not prefs.allows(intent.recipient_id,
                        intent.event_type, channel):
        return record(key, "suppressed")  # checked at
                                          # SEND time
    limiter.acquire(channel)              # provider cap
    try:
        provider.send(render(intent, channel))
        record(key, "sent")               # not
    except Transient:                     # "delivered"
        retry_with_backoff(key)

What we're doing: Follow one intent through preference checks, a rate-limited provider, a retry and a late bounce.

delivery-trace.txttext

intent 4471  order.shipped  recipient u_92

fan-out at send time, against current prefs:
  email  -> allowed
  push   -> allowed
  sms    -> SUPPRESSED (u_92 opted out at 08:14,
            after the intent was recorded at 08:12)

email delivery (key 4471:email)
  09:00:00  claim ok -> queued
  09:00:01  limiter: 500/min bucket empty, wait
  09:00:04  provider 429 -> transient, backoff
  09:00:12  provider 202 accepted -> sent
  09:03:40  webhook: bounce, 550 mailbox
            unavailable -> bounced
            address added to suppression list

push delivery (key 4471:push)
  09:00:00  claim ok -> queued
  09:00:02  provider 200 -> sent
  09:00:09  webhook: delivered -> delivered

worker crashes and the queue redelivers 4471
  09:05:00  claim(4471:email) -> already claimed,
            no second email is sent

6: The opt-out happened two minutes after the intent was recorded. Checking preferences at send time rather than at intent time is what makes that opt-out effective — the alternative sends a message the recipient has already refused.
13: A 429 is a transient failure, so it is retried with backoff. It is also a signal to the limiter: repeatedly hitting the provider's cap means your own rate limit is set too high, not that the provider is unreliable.
15: The message was accepted at 09:00:12 and bounced at 09:03:40. Any design that records "sent" as the final state reports this as a success forever, which is why acceptance and delivery need separate states and a webhook to move between them.
24: Deduplication is keyed on intent id plus channel, so a queue redelivery finds the key claimed and stops. Keying on the attempt instead would treat the redelivery as new work, which is the standard way at-least-once queues produce duplicate emails.

Why this works: Each of the five requirements shows up here doing one job — the queue absorbs the 429 wait, the state machine separates accepted from delivered, backoff handles the transient error, the claim key stops the redelivery, and the preference check at send time honours a two-minute-old opt-out. A design missing any one of them fails this exact trace in a way the recipient notices.

Recording "sent" as success and never listening for delivery events

Wrong

python

resp = provider.send(message)
if resp.status == 202:
    delivery.state = "delivered"   # it is not
                                   # delivered; it is
                                   # accepted

Better

python

resp = provider.send(message)
if resp.status == 202:
    delivery.state = "sent"        # awaiting the
                                   # delivery event
# a webhook later moves it to delivered or bounced

What you see: Your dashboard shows a 99.9% delivery rate while support handles a steady stream of customers who never received anything. Bounces are invisible, so bad addresses are retried forever and sender reputation degrades until a whole provider starts rejecting your mail.

Why: Providers accept a message for delivery and attempt it afterwards, so the API response reports acceptance, not arrival. Treating the two as one collapses the only signal that distinguishes a working address from a dead one, and it is that signal — the bounce event — that a suppression list has to be built from.

The delivery state machine for one channel attempt

queued

start

suppressed

end

sent (provider accepted)

delivered

end

bounced

end

failed (retries exhausted)

end

Delivery states and what moves a message between them

Delivery states and what moves a message between them
State	Meaning	Moves on
queued	Accepted for delivery, not yet attempted	A worker picks it up
suppressed	Never attempted — preference, quiet hours, or a prior hard bounce	Terminal
sent	The provider accepted the request	A delivery-event webhook, or nothing (unknown)
delivered	The provider confirms it reached the recipient	Terminal
bounced	Permanently undeliverable — bad address, blocked	Terminal; add the address to a suppression list
failed	Transient error, retries exhausted	Terminal; dead-lettered for inspection

Transient versus permanent, and what each deserves

Transient versus permanent, and what each deserves
Failure	Class	Correct response
429 rate limited	Transient	Back off and retry; also slow the sender
503 / timeout	Transient	Retry with exponential backoff and jitter
550 mailbox unavailable (hard bounce)	Permanent	Do not retry; suppress the address
Unregistered device token	Permanent	Do not retry; delete the token
Recipient opted out	Not a failure	Suppress before attempting, not after

Remember: Queue the sends so a slow provider becomes queue depth, not blocked requests. Track state per attempt and keep `sent` (provider accepted) distinct from `delivered` (a webhook confirmed it) — bounces arrive minutes later. Retry transient failures with backoff and never retry hard bounces. Deduplicate on intent id plus channel so an at-least-once redelivery sends nothing twice. And check both limits at send time: preferences decide whether you may send, provider rate limits decide how fast.