beginnerintermediateadvanced

System Design · Section 74

Failure Mode Analysis

Level: intermediate
Read: 12 min
Concepts: 2

A single vague "what if it fails" question collapses seven meaningfully distinct failure modes into one, which is why a real failure-mode review runs a fixed checklist against every component instead: what if it times out, returns invalid data, goes down entirely, duplicates work, becomes slow without failing outright, loses data that was acknowledged as written, or becomes unreachable due to a network partition — each of these looks different to a caller and needs a different mitigation, and a design that only plans for one of them (usually "goes down," the loudest and most obvious case) has quietly left the other six unhandled. The second half of failure-mode analysis is recognizing that total failure — where every request or replica fails the same way — is the easy case, uniform and simple to detect and design for, while partial failure — some replicas healthy and others not, some items in a batch succeeding and others erroring, one region degraded while others are fine, a multi-step operation failing partway through — is both more common in a real distributed system and structurally harder, because the correct response depends on exactly which part failed and what state it left behind. A design built only around total failure typically assumes an operation is all-or-nothing when it is not, which either crashes on that false assumption or silently produces an inconsistent result; designing for partial failure means recording per-item or per-step outcomes explicitly and retrying only the failed parts, and monitoring the partial-failure rate directly rather than trusting an aggregate pass/fail signal that a small, real, ongoing failure rate will not move at all.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Ask all seven failure-mode questions — times out, invalid data, down, duplicates work, slow, loses data, unreachable — for every component, not just a vague "what if it fails."
"Goes down" and "becomes unreachable" are different failure modes with different observable symptoms (fast connection-refused vs. hang-until-timeout).
"Returns invalid data" is often more dangerous than an outright error, because nothing signals the caller to treat a successful-looking response with suspicion.
Total failure is uniform and easy to detect; partial failure (some succeed, some do not) is more common and needs an explicit design, not an assumption of all-or-nothing.
Record per-item or per-step outcomes in any batch or multi-step operation, and monitor the partial-failure rate directly — an aggregate exit-code or health-check signal will not move for a small, ongoing partial failure.

What you will be able to do

Run the seven-question failure-mode checklist against a given component and name a distinct mitigation for each
Explain why "goes down" and "becomes unreachable" require different handling despite both being outages
Distinguish total failure from partial failure and identify which one a given design has actually planned for
Design a batch or multi-step operation that records per-item outcomes rather than assuming all-or-nothing

The seven questions

A fixed, specific checklist that surfaces failure modes a single vague "what if it fails" question does not.

The seven-question failure-mode checklist

coreintermediate

Failure mode analysis is the practice of deliberately asking, for every component in a design, a fixed list of "what if" questions rather than only designing for the case where everything works — most design reviews default to describing the happy path in detail and mentioning failure handling as an afterthought, if at all. The seven questions are concrete and apply to almost any component: what if it times out (the caller waits and gets nothing back in time)? What if it returns invalid data (a malformed or logically wrong response, not an error)? What if it goes down entirely (hard failure, connection refused)? What if it duplicates work (the same operation runs twice, e.g. from a retry)? What if it becomes slow rather than failing outright (a partial, harder-to-detect degradation)? What if it loses data (an acknowledged write that never actually persisted)? And what if it becomes unreachable (a network partition, distinct from the component itself being down)? Running through all seven for a given component surfaces failure modes that a single generic "what if it fails" question does not, because "it fails" collapses seven meaningfully different situations — each needing a different mitigation — into one vague case that is easy to wave away with "we'll add error handling."

Think of it as

Think of a pre-flight checklist a pilot runs before every takeoff: it is not "check that the plane works," which is too vague to act on, it is a fixed list of specific, separately-checked items — fuel, flaps, instruments, communications — precisely because a vague check gets skipped or rushed while a specific one gets actually verified. The seven failure-mode questions are the same kind of checklist applied to a system component: "what if it fails" is the vague version everyone already nominally does; "what if it times out, specifically, versus what if it returns wrong data, specifically" is the version that produces a concrete answer for each case rather than a shrug.

text

# Run against every component in a design —
# here, a payment-gateway dependency:
1. Times out?          -> caller hangs past 3s -> set timeout
2. Invalid data?        -> charge succeeds but amount is wrong
                           -> validate response fields before use
3. Goes down?           -> connection refused -> circuit breaker
4. Duplicates work?     -> a retried charge double-bills the
                           customer -> idempotency key per attempt
5. Becomes slow?        -> P99 climbs to 8s without erroring
                           -> timeout catches this too
6. Loses data?          -> charge succeeds, confirmation write to
                           our DB fails -> reconcile via webhook
7. Unreachable?         -> network partition to the gateway's
                           region -> same as goes-down handling

What we're doing: Apply all seven questions to one component — an inventory service called during checkout.

inventory-service-failure-modes.txttext

Component: inventory-service.reserve_stock()

1. Times out?      -> checkout hangs; need an explicit deadline
2. Invalid data?    -> returns "reserved: true" for an item that
                       is actually out of stock (a logic bug, not
                       a crash) -> oversold inventory, no error
                       anywhere to catch it
3. Goes down?       -> checkout cannot confirm stock at all
4. Duplicates work? -> a retried reservation call double-reserves
                       the same unit of stock against two orders
5. Becomes slow?     -> checkout's overall latency climbs even
                       though nothing "fails"
6. Loses data?       -> reservation succeeds internally but the
                       confirmation event to checkout is dropped
7. Unreachable?      -> network partition between checkout's
                       region and inventory's region

3: This is the failure mode most designs miss entirely, because nothing about it looks like a failure — no timeout, no error, no down connection, just a wrong answer returned successfully, which is exactly why it needs an explicit question rather than being caught incidentally by handling for "the call fails."
5: This is a direct, concrete instance of the retry-amplification and idempotency risk covered in full in Retry Strategy and Delivery Semantics — the checklist's job here is just to surface that this component needs that treatment, not to re-derive the mechanism.

Why this works: A review that only asks "what if inventory-service fails" produces a single fallback plan (probably: fail the checkout), which is the right answer for exactly one of these seven cases (goes down) and the wrong or incomplete answer for the other six — running the full checklist is what turns one vague plan into seven specific, correct ones.

Designing failure handling only for "the service is down"

Wrong

python

def reserve_stock(order):
    try:
        return inventory_service.reserve(order)
    except ConnectionError:
        raise CheckoutFailed("inventory unavailable")
    # no timeout, no idempotency key, no response
    # validation -- only the "down" case is handled

Better

python

def reserve_stock(order):
    try:
        result = inventory_service.reserve(
            order, idempotency_key=order.id, timeout=2.0)
    except (ConnectionError, Timeout):
        raise CheckoutFailed("inventory unavailable")
    if not result.is_valid():
        raise CheckoutFailed("invalid reservation response")
    return result

What you see: The incident review after an oversold-inventory event finds that the code has a `try/except ConnectionError` and nothing else — no timeout, no idempotency key, no validation of the response — because "handle the failure" was interpreted narrowly as "handle the service being down," which was the only failure mode anyone explicitly designed for.

Why: ConnectionError is the easiest failure mode to imagine because it is the loudest and most obviously a failure — the other six modes either look like success (invalid data), only show up under retry conditions (duplicated work), or only show up under load (becomes slow), so a design that stops at "handle the down case" has quietly skipped six-sevenths of the checklist without anyone deciding to.

Seven questions to ask about every component

Times out

caller waits, gets nothing back in time

Returns invalid data

a response arrives, but is wrong

Goes down

hard failure, connection refused

Duplicates work

same operation runs twice

Becomes slow

degraded, not failing outright

Loses data

an acknowledged write never persists

Becomes unreachable

network partition, not the same as down

The seven failure modes and a typical mitigation for each

The seven failure modes and a typical mitigation for each
Failure mode	What it looks like to the caller	Typical mitigation
Times out	No response within an expected window	Timeout + retry with backoff
Returns invalid data	A response arrives, but is wrong or malformed	Response validation / schema checks
Goes down	Immediate connection failure	Circuit breaker, fallback
Duplicates work	The same operation is applied twice	Idempotency keys / idempotent consumers
Becomes slow	Degraded but not failing latency	Timeouts, load shedding, bulkheads
Loses data	An acknowledged write does not persist	Durable writes, replication, write acknowledgment semantics
Becomes unreachable	Network partition, distinct from being down	Timeouts, retries, partition-aware design

Remember: Ask all seven questions — times out, returns invalid data, goes down, duplicates work, becomes slow, loses data, becomes unreachable — for every component in a design, not just the ones that feel risky, and not collapsed into one vague "what if it fails." Each question surfaces a distinct failure mode with a distinct mitigation; a design that only plans for "goes down" has answered one of seven questions.

Total failure vs. partial failure

Why the more common case — some parts failing while others succeed — needs its own explicit design, not an all-or-nothing assumption.

Designing for partial failure, not just total failure

coreintermediate

Total failure — a service is completely down, every request fails the same way — is the easy case to design for, because the correct behavior is simple and uniform: fail fast, alert, fail over. Partial failure is far more common in a real distributed system and much harder to design for correctly: some requests succeed and some fail (one replica is unhealthy, others are fine), some fraction of a batch job's items succeed and others error, one region is degraded while others are healthy, or a multi-step operation succeeds on its first three steps and fails on the fourth, leaving the first three "done" with no clean way to undo them. A design that only handles total failure ("if the service is down, do X") has no defined behavior for these in-between states, so the code either crashes on an assumption that does not hold (assuming a batch either fully succeeds or fully fails) or silently produces an inconsistent result (some of a user's items got processed, some did not, and nothing recorded which). Designing for partial failure means explicitly deciding, for every multi-part or multi-replica operation, what "some succeeded and some did not" actually means for correctness — is it retried as a whole, retried only for the failed parts, rolled back, or left as a partial success that gets reconciled later?

Think of it as

Total failure is a power outage — the whole building goes dark at once, and the response (evacuate, wait for power) is the same regardless of which room you are in. Partial failure is more like one floor of the building losing power while the rest stays lit — the correct response depends entirely on which floor you are on, what was happening on it when the power went, and whether anything on that floor was mid-task (an elevator stuck between floors is a very different problem than a dark hallway with no one in it). A building designed only for the "total blackout" scenario has an evacuation plan but no idea what to do about the stuck elevator, because nobody designed for the case where only part of the building fails.

python

# A batch operation designed for total failure only
# (assumes all-or-nothing):
def process_batch(items):
    for item in items:
        process(item)          # any exception aborts
                                # the whole batch, or is
                                # silently swallowed --
                                # neither records which
                                # items actually succeeded

# Designed for partial failure:
def process_batch(items):
    results = {"succeeded": [], "failed": []}
    for item in items:
        try:
            process(item)
            results["succeeded"].append(item.id)
        except ProcessingError as e:
            results["failed"].append((item.id, str(e)))
    persist_batch_result(results)   # failed items are
                                     # known and retryable
    return results

What we're doing: Trace what happens to a 1,000-item nightly export job when 12 items fail partway through.

export-job-outcomes.txttext

Design A (total-failure-only):
  job processes items 1-987 successfully, item 988
  throws an unhandled exception -> job crashes -> items
  989-1000 never attempted, and there is no record of
  which of 1-987 actually completed vs. which succeeded
  before the crash

Design B (partial-failure-aware):
  job processes all 1,000 items, records 988 successes
  and 12 named failures (item IDs + error reasons),
  completes normally, and the 12 failures are queued
  for individual retry

3: The crash at item 988 is the total-failure design leaking through: one bad item is treated as if it were the same class of event as the whole service being down, aborting work that had nothing to do with the actual problem.
9: The partial-failure design treats one item's failure as a fact to record, not an event that should propagate up and stop unrelated work — the other 988 items succeed regardless, and the 12 failures are individually actionable instead of an unknown quantity somewhere in a crashed run.

Why this works: Design A's failure mode is not "the job does not work" — it visibly crashes and gets noticed — but the operational cost of not knowing exactly which items succeeded before the crash is real: re-running the whole job risks duplicate side effects for the 987 items that already succeeded, unless every one of those side effects also happens to be idempotent, which is not something to assume by default.

Letting one item's exception abort processing for every other item in a batch

Wrong

python

def send_notifications(users):
    for user in users:
        send_email(user)   # one bad email address
                            # raises, killing the loop
                            # for every remaining user

Better

python

def send_notifications(users):
    failed = []
    for user in users:
        try:
            send_email(user)
        except EmailError as e:
            failed.append((user.id, str(e)))
    if failed:
        log_and_queue_for_retry(failed)

What you see: A nightly notification job silently stops sending to the remaining 40% of users every time it hits one user with an invalid email address, and nobody notices for weeks because the job "completes" (it just stops early) without an error loud enough to page anyone.

Why: A loop with no per-item error boundary treats every item's failure as equivalent to a total failure of the whole operation — the correct scope for the failure (one user's bad email address) is far smaller than the scope the code actually gives it (every user after that point in the loop), and nothing in the design ever draws that line explicitly.

A batch job: designed for total failure vs. partial failure

Designed for total failure only

+Any single item error aborts or is silently swallowed
+No record of which specific items succeeded or failed
+Retrying means reprocessing the whole batch, including items that already succeeded
+Monitoring only alerts if the whole job crashes

Designed for partial failure

−Each item's outcome is recorded independently
−Failed items are known by ID and retryable on their own
−Succeeded items are never reprocessed
−A rising failed-item rate is visible even while the job "succeeds" overall

Total vs partial failure: the same component, two different design problems

Total vs partial failure: the same component, two different design problems
Aspect	Total failure	Partial failure
What happened	Every request fails the same way	Some requests/items/replicas fail, others succeed
Detection	Easy — aggregate health check trips	Hard — aggregate metrics can look healthy overall
Correct response	Uniform: fail fast, alert, fail over	Depends on which part failed and what state it left behind
Common design gap	Usually well-covered by monitoring/alerting	Often unhandled — code assumes all-or-nothing

Remember: Total failure (uniform, easy to detect, easy to design for) and partial failure (some succeed, some do not — some replicas, some batch items, some steps of a multi-step operation) are different design problems, and a design that only handles the first has no defined behavior for the much more common second. Record per-item/per-part outcomes explicitly, retry only the failed parts, and monitor the partial-failure rate directly rather than only the aggregate success/fail signal.