beginnerintermediateadvanced

System Design · Section 82

Payment System Design

Level: advanced
Read: 20 min
Concepts: 3

A payment flow is the canonical workflow you cannot make atomic: your database and the payment provider share no transaction, and money has often already moved by the time you learn whether your own write succeeded. Every request therefore has three outcomes rather than two — it worked, it failed, or you do not know — and that third outcome is the whole design problem. Six building blocks make it recoverable. An idempotency key, generated before the first attempt and reused on every retry, lets the provider recognise a repeated charge as the same charge instead of a second one. Immutable transaction records append what happened rather than updating a current value, so history stays readable and reconcilable. Explicit payment states replace a boolean `paid` flag, giving the in-between state — where asynchronous methods spend real time — somewhere to live. Webhook processing carries outcomes that arrive after the customer has left, and must be written for events that are redelivered, arrive out of order, and sometimes reference a payment your own write has not committed yet. Reconciliation is a scheduled comparison against the provider's records, designed on the assumption that discrepancies exist. Together these produce retry safety, the property the whole flow depends on. Two further rules follow. A browser redirect to your success URL is not proof of payment — it can be forged by anyone who types the URL, missed entirely by a customer who closed the tab, or arrive while an asynchronous charge is still pending — so it decides only what to display, while fulfilment hangs off a signature-verified webhook or a server-to-server lookup. And the provider-owned payment lifecycle and the fulfilment lifecycle you own progress independently, so they belong in two state machines with an explicitly enforced link, rather than one status column whose value count grows as the product of both.

This section

Design a Payment System - System Design InterviewCode with Lucian

What is true here

Every provider request has three outcomes — worked, failed, unknown — and the unknown one is what the six building blocks exist to make recoverable.
An idempotency key identifies a logical operation, not an attempt: generate it before the first try and reuse it on every retry.
Webhooks arrive more than once, out of order, and sometimes before your own write is visible — dedupe on the event id, claimed before the effect is applied.
A browser redirect can be forged, missed, or premature; fulfilment must hang off a signature-verified webhook or a server-to-server lookup.
Payment state and fulfilment state are independent lifecycles; one status column has to enumerate their product, two columns enumerate their sum.

What you will be able to do

Design a charge path that stays correct when the provider call times out with an unknown outcome
Process webhooks that are redelivered, reordered, or reference a payment you have not recorded yet
Explain the three distinct ways a success redirect misleads, and what to use instead for fulfilment
Model payment and fulfilment as two state machines with an enforced link, and query the combinations that matter operationally

Making a payment flow correct

Six building blocks, each closing a gap the others leave open.

Idempotency keys, immutable records, explicit states, webhooks, reconciliation

coreadvanced

A payment flow is the standard example of a workflow you cannot make atomic: your database and the payment provider are two systems with no shared transaction, and money has already moved by the time you find out whether your own write succeeded. Six building blocks handle that, and they are not independent options — each one closes a gap the others leave open. An idempotency key, sent by you with the charge request, lets the provider recognise a repeated request as the same request and return the original result instead of charging again, which is what makes a retry after a timeout safe. Immutable transaction records mean you append what happened rather than updating a row in place, so a refund is a new row and the original charge is still readable — an audit trail you can reconcile against, rather than a current value you have to trust. Explicit payment states (`requires_payment_method`, `processing`, `succeeded`, `failed`) replace a boolean `paid` flag, so the very common in-between state has a name and the code has somewhere to put it. Webhook processing is how you learn the outcome of anything asynchronous, because a payment can complete minutes after the customer closed the tab. Reconciliation is a scheduled job that compares your records against the provider's and reports the differences, on the assumption that some will exist. And retry-safe operations means every step in the flow can run twice without a second effect — which is what the previous five blocks together are for.

Think of it as

Treat the payment provider as a system you can send instructions to and never fully observe. Every request you make has three possible outcomes, not two: it worked, it failed, or you do not know. That third outcome is the entire design problem, and it is not rare — a timeout, a dropped connection, or a deploy mid-request all produce it. The six building blocks exist so that "I do not know" is always recoverable: the idempotency key lets you ask again safely, the explicit state gives the unknown a name, the webhook eventually tells you, the immutable log records what you learned and when, and reconciliation catches the cases where none of that worked. A payment system is not code that charges cards; it is a bookkeeping system that happens to charge cards.

http

POST /v1/payment_intents HTTP/1.1
Host: api.stripe.com
Idempotency-Key: order_8814_attempt_1
Content-Type: application/x-www-form-urlencoded

amount=4200&currency=usd&customer=cus_...

# Same key again -> the original PaymentIntent is
# returned. No second charge is created.

What we're doing: Trace one order through an ambiguous timeout, a duplicate webhook, and the reconciliation job that checks the result.

payment-trace.txttext

t0   INSERT payment_events
       (order_id=8814, type='intent_created',
        state='processing')

t1   POST /charges  Idempotency-Key: order_8814
       -> TCP timeout after 30s. Outcome unknown.
       Local state stays 'processing'. Nothing is
       marked failed, because nothing is known.

t2   Retry POST /charges, same idempotency key
       -> 200 {id: ch_9f2, status: succeeded}
       Provider recognised the key; one charge
       exists, not two.

t3   INSERT payment_events
       (order_id=8814, type='captured',
        provider_id='ch_9f2', state='succeeded')

t4   Webhook payment_intent.succeeded (evt_a1)
       -> already applied? no. apply, record evt_a1.

t5   Webhook payment_intent.succeeded (evt_a1)
       AGAIN (provider retried, our 200 was lost)
       -> evt_a1 already recorded. Ignore. 200.

t6   Nightly reconciliation
       provider charges for the day: 1,204
       local 'succeeded' events for the day: 1,203
       -> 1 discrepancy, reported, investigated

6: This is the state the whole design exists for. The request neither succeeded nor failed from your side, and marking it failed here is the single most common way to charge a customer for an order you then tell them did not go through.
10: The retry is safe only because of the idempotency key. Without it this exact line is a second charge, and the customer sees two identical amounts on their statement.
22: Deduplication is on the provider's event id, recorded before the effect is applied — so a redelivery finds the id already present and does nothing. Deduplicating on order id instead would wrongly discard a genuinely different later event for the same order.
28: The reconciliation job is designed around finding a difference, not around proving there is none. One discrepancy in 1,204 is a normal day; the job exists so that the number is known and investigated rather than discovered by an accountant a quarter later.

Why this works: Every one of the six blocks appears in this trace doing exactly one job: the explicit state gives t1's unknown outcome somewhere to live, the idempotency key makes t2 safe, the immutable event rows at t0/t3 make the history readable, webhook dedupe handles t5, and reconciliation at t6 catches what the first five missed. Remove any one and the trace has a hole a real customer eventually falls into.

Generating a fresh idempotency key on each retry

Wrong

python

for attempt in range(3):
    key = str(uuid.uuid4())   # new key per attempt
    try:
        return provider.charge(amount, idem_key=key)
    except Timeout:
        continue

Better

python

key = f"order-{order_id}-charge"   # stable per
for attempt in range(3):          # logical operation
    try:
        return provider.charge(amount, idem_key=key)
    except Timeout:
        continue

What you see: A customer is charged twice or three times for one order, and only during periods when the provider was slow. The logs show a successful charge on the third attempt and no error at all, because from the code's point of view the first two attempts genuinely did time out.

Why: An idempotency key identifies a logical operation, not an attempt. A fresh key on each retry tells the provider "this is a new charge", which is precisely the opposite of what a retry means — the key must be derived from something stable about the operation, and it must be generated before the first attempt, not inside the retry loop.

A charge, an ambiguous timeout, and a safe retry

Your service

Provider

Your database

1. record intent (state: processing)
2. POST charge, Idempotency-Key: order_8814
3. connection times out — outcome unknownmoney may or may not have moved
4. POST charge again, SAME idempotency key
5. original charge returned, not a second one
6. webhook: payment_intent.succeededmay arrive before or after the retry
7. append event, advance state to succeeded

Six building blocks and the specific failure each one closes

Six building blocks and the specific failure each one closes
Block	Failure it closes	What happens without it
Idempotency key	A retry after a timeout on a charge whose result you never saw	The customer is charged twice; you find out via a chargeback
Immutable transaction records	A dispute about what happened and in what order	The current row says `refunded` and nothing says what the original charge was
Explicit payment states	A payment that is neither clearly paid nor clearly failed	The in-between state is encoded as "not paid", so the customer is asked to pay again
Webhook processing	An outcome that arrives after the customer left	Asynchronous methods appear to fail; delayed successes are never recorded
Reconciliation	Everything the first four blocks still missed	Discrepancies accumulate silently until an accountant finds them
Retry-safe operations	Any step re-running after a partial failure	A resumed workflow duplicates whichever steps already completed

Three webhook properties you must design for, not hope against

Three webhook properties you must design for, not hope against
Property	Concrete case	Required handling
Delivered more than once	The provider retries because your 200 response was lost	Dedupe on the event id before applying any effect
Delivered out of order	`payment.succeeded` arrives after `charge.refunded`	Apply by state machine and event version, never by arrival order
Delivered before your own write is visible	The webhook lands before your create-order transaction commits	Look up by the provider's id; if unknown, record and retry rather than discard

Remember: Six blocks, each closing a gap the others leave: idempotency keys make retrying an ambiguous timeout safe; immutable event rows keep history reconcilable; explicit states give the in-between outcome a name; webhook processing carries asynchronous results, deduplicated on the provider's event id and claimed before the effect is applied; reconciliation is a scheduled job that assumes discrepancies exist; and retry safety is the property all five together produce. The hard case is not failure — it is not knowing.

Trusting the right signal

Why the redirect is not the outcome, and what is.

A frontend redirect is not proof that a payment completed

coreadvanced

After a customer pays on a provider's hosted page, their browser is redirected back to a success URL on your site. That redirect is a navigation event in a browser you do not control, and it is not a statement from the provider that money moved. Three things are wrong with treating it as one. It can be forged: the URL is visible in the address bar, and anyone can type it, bookmark it or share it, so a request to `/checkout/success?order=8814` proves only that someone requested that path. It can be missed: the customer closes the tab, loses signal, or the payment method completes asynchronously minutes later, so a genuinely successful payment produces no redirect at all. And it can arrive before the payment is final: some methods redirect while the charge is still pending, so the redirect is truthful about "the customer finished their part" and silent about "the money settled". The correct design uses the redirect for exactly one thing — deciding what to show the customer next — and takes the actual payment outcome from a server-side source: a webhook from the provider, or a server-to-server lookup of the payment by its id. Fulfilment, entitlement and receipts hang off that source, never off the redirect.

Think of it as

The redirect is the customer telling you they think they paid. The webhook is the provider telling you they did. Those are different witnesses with different reliability, and you would not ship a physical package on the strength of the first one. Treat the success page as a courtesy — a place to say "thanks, we are confirming your payment" — and treat the server-side signal as the thing that actually unlocks anything. When the two disagree, the server-side signal wins, every time, without exception.

python

# The success route decides what to SHOW, nothing else
@app.get("/checkout/success")
def success(payment_intent_id: str):
    # server-to-server, not the query string
    intent = provider.retrieve(payment_intent_id)
    if intent.status == "succeeded":
        return render("thanks.html")
    return render("confirming.html")   # webhook will
                                       # settle it

What we're doing: Compare what a redirect-trusting checkout and a webhook-driven checkout do across four real situations.

redirect-vs-webhook.txttext

Case 1  Customer pays, redirect arrives
  trusts redirect : order fulfilled            OK
  webhook-driven  : order fulfilled            OK

Case 2  Customer pays, closes tab before redirect
  trusts redirect : order never fulfilled;
                    customer charged, gets nothing
  webhook-driven  : webhook fulfils it minutes
                    later                      OK

Case 3  Someone types /checkout/success?order=8814
  trusts redirect : order fulfilled, no payment
  webhook-driven  : nothing happens; the page
                    shows "confirming"          OK

Case 4  Async method: redirect now, settles in 2h
  trusts redirect : fulfilled immediately, then
                    the payment fails
  webhook-driven  : stays 'processing'; fulfils
                    on payment_intent.succeeded  OK

6: This is the case teams discover last and lose the most money to, because it looks like a bug in the payment provider rather than a design choice: the money moved, and the only signal the design listens to never arrived.
12: The forged-URL case is the one security review catches. It is also the least common in practice — which is why a design that only defends against this one still fails cases 2 and 4.
17: An asynchronous method makes the redirect honest and still wrong: the customer did finish their part. Fulfilling here means shipping goods against a payment that can still fail, with no way to tell the difference from a successful one at redirect time.

Why this works: The redirect is correct in exactly one of these four cases and misleading in three, in three different ways — absent, forged, and premature. A webhook-driven design gets all four right because it takes the outcome from the party that actually knows it, and uses the redirect only to decide which page to render.

Fulfilling the order in the success route

Wrong

python

@app.get("/checkout/success")
def success(order_id: str):
    order = db.get(order_id)
    order.status = "paid"      # trusting a URL
    fulfil(order)              # ship it
    return render("thanks.html")

Better

python

@app.post("/webhooks/provider")
def webhook(request):
    event = provider.verify(request.body,
                            request.headers["Signature"])
    if event.type == "payment_intent.succeeded":
        if claim_event(event.id):      # dedupe first
            mark_paid_and_fulfil(event.data.order_id)
    return 200

What you see: Free orders. Someone shares or guesses the success URL and receives goods or a subscription with no payment attached. In parallel, a steady trickle of paying customers who closed the tab receive nothing, and contact support with a bank statement showing the charge.

Why: The success route is reachable by anyone who can type a URL and unreachable by any customer whose browser did not complete the round trip, so it is both too permissive and too strict. The webhook is signed, server-to-server, retried until acknowledged, and independent of the customer's browser — which is exactly the set of properties fulfilment needs.

Two paths carry two different kinds of truth

Customer pays on hosted page

Browser redirect to /success

forgeable, skippable, possibly early

Decide what to display

thanks, or "confirming your payment"

Signed webhook to your server

the authoritative outcome

Server-to-server lookup by id

same authority, pull instead of push

Fulfil, grant entitlement, send receipt

Three signals, and what each actually proves

Three signals, and what each actually proves
Signal	Proves	Does not prove
Redirect to your success URL	A browser requested that URL	That any payment was attempted, let alone completed
Signed webhook from the provider	The provider says this payment reached this state	That you have processed it — you still dedupe and record it
Server-to-server lookup by payment id	The payment's state at the moment you asked	That it will not change later (refunds, disputes, async settlement)

Remember: The redirect tells you the customer thinks they paid; the webhook tells you the provider says they did. A redirect can be forged, missed entirely, or arrive while the charge is still pending, so use it only to decide what to display. Fulfilment, entitlement and receipts hang off a signature-verified webhook or a server-to-server lookup by payment id — and an unverified webhook endpoint is a forgeable URL exactly like the redirect it replaced.

Modelling the lifecycles

Payment and fulfilment as two independent state machines with one written-down link.

Separating payment state from order and fulfilment state

coreadvanced

A single `status` column on an order tries to hold two different lifecycles at once, and they do not line up. The payment lifecycle belongs to the provider and moves through states like requires-payment-method, processing, succeeded, failed, refunded, disputed. The fulfilment lifecycle belongs to you and moves through pending, allocated, packed, shipped, delivered, returned, cancelled. These progress independently: a payment can succeed while fulfilment has not started, fulfilment can be underway while a dispute is opened, and a refund can happen long after delivery. Squashing both into one enum forces you to invent hybrid values — `paid_but_not_shipped`, `shipped_refund_pending` — and the count of those values grows as the product of the two lifecycles, not their sum. Two state machines with an explicit link between them keeps each one small and independently correct, lets each move at its own speed, and makes the interesting questions answerable with a query: which orders are paid and unfulfilled, which are fulfilled and unpaid. The link is a rule you write down — typically "fulfilment starts when payment reaches succeeded" — rather than a shared column that pretends the two lifecycles are one.

Think of it as

Think of a restaurant. The kitchen has a ticket that moves from ordered to cooking to plated to served. The till has a bill that moves from open to paid to, occasionally, refunded. Nobody tries to run both on one piece of paper, because the food and the money genuinely move at different times and sometimes in different directions — a meal can be comped after it is eaten, and a bill can be settled before the food arrives. The two tickets reference each other by table number. Your order and your payment work the same way, and the "table number" is the order id.

sql

-- two columns, two machines, one link rule
CREATE TABLE orders (
  id            uuid PRIMARY KEY,
  payment_state text NOT NULL,   -- provider's machine
  fulfil_state  text NOT NULL,   -- yours
  CHECK (fulfil_state = 'pending'
         OR payment_state = 'succeeded')
);

-- the question a single enum cannot answer cleanly
SELECT id FROM orders
 WHERE payment_state = 'disputed'
   AND fulfil_state IN ('shipped', 'delivered');

What we're doing: Watch one order move through both machines and see the moments where they are out of step.

two-machines-trace.txttext

                payment_state    fulfil_state
checkout        processing       pending
webhook ok      succeeded        pending
picker starts   succeeded        allocated
handed to DHL   succeeded        shipped
delivered       succeeded        delivered
day 14: chargeback opened
                disputed         delivered
day 21: dispute lost
                refunded         delivered
customer returns the item
                refunded         returned

Every row above is a legitimate state of a real
order. A single enum would need a distinct value
for each of the 9 rows, and for every other path
through the two machines.

3: Payment moves first and fulfilment has not started — the ordinary state of every order in the seconds or minutes after checkout, and the one a `paid` boolean cannot distinguish from "delivered".
8: The payment machine moves backward in customer-value terms while the fulfilment machine stays put. Neither machine has an invalid transition here; the pair is just uncomfortable, which is exactly the situation operations needs to query for.
12: The two machines finish in different orders than they started. Only independent state can represent this without inventing a combined value nobody thought of when the enum was written.

Why this works: The trace makes the multiplicative growth concrete: nine rows here are nine combinations, and this is one path through two small machines. Separating them means each transition is validated against its own machine — a payment cannot go from `failed` to `refunded`, a fulfilment cannot go from `pending` to `delivered` — while the pair is left free to be whatever the real world produced.

Deriving fulfilment progress from the payment state

Wrong

python

def is_shipped(order):
    return order.status in (
        "paid_and_shipped",
        "paid_shipped_refund_pending",
        "paid_shipped_disputed",
    )   # a new payment state adds three more values

Better

python

def is_shipped(order):
    return order.fulfil_state in ("shipped", "delivered")
# unaffected by anything the payment machine does

What you see: A new payment state is added — a dispute, a partial refund — and shipping labels stop printing for affected orders, because a list of enum values somewhere else in the codebase was never updated to include the new combinations.

Why: When one column encodes two lifecycles, every predicate about one lifecycle has to enumerate the other, so adding a state to either side silently invalidates predicates across the whole codebase. Two columns make each predicate depend only on the machine it is asking about.

One status column versus two state machines

One `status` enum

+Values must cover every payment × fulfilment pair
+Hybrid names appear: paid_but_not_shipped, shipped_refund_pending
+Adding one payment state forces a new value per fulfilment state
+"Paid and unfulfilled" needs a list of specific enum values, kept in sync by hand

Two state machines

−Each machine has its own small, testable transition set
−Adding a payment state adds one value, not a row of combinations
−Independent progress is representable, because it is representable in the data
−"Paid and unfulfilled" is a two-column WHERE clause

Two state machines, one link

Two state machines, one link
Machine	Owned by	States
Payment	The payment provider; you observe it	requires_payment_method · processing · succeeded · failed · refunded · disputed
Fulfilment	You; the provider knows nothing about it	pending · allocated · packed · shipped · delivered · returned · cancelled
Link	A rule you write down and enforce	fulfilment leaves `pending` only when payment is `succeeded`

Combinations that a single-enum design has no clean value for

Combinations that a single-enum design has no clean value for
Payment	Fulfilment	What it means operationally
succeeded	pending	Money taken, nothing allocated yet — the normal state right after checkout
processing	pending	Asynchronous method still settling; do not ship, do not cancel
succeeded	shipped	The ordinary completed order
disputed	shipped	Goods are gone and the money is being clawed back — the case you most need to query for
refunded	delivered	A post-delivery refund; the return may or may not follow

Remember: Payment state belongs to the provider, fulfilment state belongs to you, and they progress independently — paid-and-unfulfilled, disputed-and-delivered, refunded-and-not-yet-returned are all normal. One `status` column has to enumerate the product of both lifecycles; two columns keep each machine small and make the operational questions plain WHERE clauses. Then write the link between them down as a constraint, because splitting the machines removes the coupling the single enum was accidentally providing.