beginnerintermediateadvanced

System Design · Section 76

Graceful Degradation

Level: intermediate
Read: 12 min
Concepts: 2

Graceful degradation only exists if a degraded mode is deliberately defined, coded, and tested for a given dependency's failure — labeling a dependency "optional" in a dependency map is a necessary first step, but changes nothing about the running system until the fallback behavior is actually built, and confirmed to produce a genuinely usable result rather than a technically-non-crashing but visibly broken one. Four patterns cover most real designs: serving stale cache trades data freshness for continued availability, appropriate when a slightly outdated value is far less damaging than none at all; disabling a feature outright removes it from view entirely when no reasonable substitute value exists, which is clearer to a user than showing a broken or empty version of it; queuing work (like a notification) lets the requesting action succeed immediately while the deferred effect completes once its dependency recovers, appropriate for anything that does not need to happen synchronously; and returning partial results commits to whatever subset of an aggregated response is actually available, appropriate when some correct data is more useful to the caller than none. Choosing the right pattern for the right situation matters as much as building one at all — stale cache is the wrong choice for data whose correctness cannot be traded away (an active balance during a funds transfer), and a partial result should be marked as partial rather than silently returned as if it were complete, since whether a response is whole or missing pieces is itself information the caller may need.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

A degraded mode is a specific, designed, and tested behavior for a dependency's failure — not whatever an unhandled exception happens to produce.
Classifying a dependency as optional does not, by itself, change the running system's behavior — the fallback code still has to be written and verified.
Four patterns cover most needs: serve stale cache, disable the feature, queue the work, return partial results.
Match the pattern to what can safely be traded away — staleness is fine for a dashboard glance, dangerous for an active balance check.
A partial result should be visibly marked as partial, not silently returned as if it were complete — completeness is itself meaningful data to the caller.

What you will be able to do

Explain why classifying a dependency as optional does not by itself create a degraded mode
Design and test a fallback behavior for a given dependency failure, not just handle the exception
Choose the correct degradation pattern (stale cache, disable, queue, partial results) for a given situation
Recognize when a degradation pattern trades away something that should not be traded away for that specific use case

Defining it deliberately

Why a degraded mode has to be designed and tested, not left to emerge from whatever an unhandled failure happens to do.

Defining a degraded mode, deliberately

coreintermediate

Graceful degradation is the practice of deciding, ahead of time and on purpose, exactly what a system can still do when a non-critical dependency fails, rather than discovering it by accident during an actual outage. This connects directly to the critical-vs-optional classification from Dependency Management: an optional dependency, by definition, is one whose failure should not stop the primary function — but "should not stop it" is not automatically true just because a dependency was labeled optional. It is only true if someone actually defined and built the specific reduced-functionality behavior for that failure case, tested it, and confirmed it produces a usable result rather than a confusing half-broken one. A degraded mode is that specific, defined behavior — not "the system tries to keep going and whatever happens, happens," but an explicit, designed answer to "if this dependency is down, this feature is disabled/reduced/replaced with a fallback, and here is exactly what the user sees instead." Systems that never do this work end up with an accidental, undefined degraded mode that emerges from whatever the code happens to do when a call fails — usually an error, a blank section, or a crash — none of which is the deliberate, still-useful reduced state that graceful degradation is aiming for.

Think of it as

A commercial airplane has a deliberately designed set of degraded modes for equipment failure — if one hydraulic system fails, there are backup systems and a defined reduced-capability flight profile, all specified in advance by engineers who asked "what can this plane still safely do with this system out" long before any specific flight. It is not the pilot improvising in the moment; the degraded behavior was designed, tested, and documented before it was ever needed. A system with no defined degraded mode is like a plane with no such engineering — when a system fails in flight, what happens next is whatever emerges from the wreckage of untested assumptions, not a deliberately chosen safe reduced state.

text

# A degraded-mode definition, one line per
# optional dependency -- the test for whether
# graceful degradation actually exists:
recommendations-service down:
  -> product page renders without the "customers
     also bought" section; no error shown
loyalty-points-service down:
  -> checkout completes normally; points awarded
     asynchronously once the service recovers
search-autocomplete down:
  -> search box still accepts full-text queries on
     submit; only the live-typing suggestions
     disappear

What we're doing: Compare what actually happens to a product page when the recommendations service fails, with and without a defined degraded mode.

product-page-no-degraded-mode.pypython

def render_product_page(product_id):
    product = product_service.get(product_id)
    upsells = recommendations_service.get_upsells(product_id)
    return render_template(product=product, upsells=upsells)
    # no handling for recommendations_service failing --
    # an exception here takes down the whole page

3: This line has no defined degraded mode — if recommendations_service raises, there is no code path that says what the page should do instead, so whatever the framework's default unhandled-exception behavior is (usually a 500 error page) becomes the accidental "degraded mode."

Why this works: Nothing about labeling recommendations-service "optional" in a dependency map changes what this code actually does when the call fails — the classification is a design intention, and without the corresponding code (and a test that actually exercises the failure), the intention and the real behavior of the system are two different things.

Classifying a dependency as optional without building its degraded mode

Wrong

python

# dependency map says: "recommendations-service:
# optional, hide the section on failure" -- but
# the code was never updated to actually do that
def render_product_page(product_id):
    product = product_service.get(product_id)
    upsells = recommendations_service.get_upsells(product_id)
    return render_template(product=product, upsells=upsells)

Better

python

def render_product_page(product_id):
    product = product_service.get(product_id)
    try:
        upsells = recommendations_service.get_upsells(
            product_id, timeout=0.5)
    except (Timeout, ServiceError):
        upsells = []
    return render_template(product=product, upsells=upsells)

What you see: A design review or dependency map correctly lists recommendations-service as "optional, degrade gracefully," and a real outage six months later still takes the whole product page down, because the classification was never followed by the actual code change and test that would have made it true.

Why: A dependency map is a document describing intended behavior; it has no effect on the running system until the corresponding fallback code is written and verified — treating the classification itself as the fix is mistaking the plan for the implementation.

From normal operation to a deliberately defined degraded mode

Normal operation

start

Dependency failure detected

Defined degraded mode active

Dependency recovers

end

Accidental vs. deliberate degraded mode for the same failure

Accidental vs. deliberate degraded mode for the same failure
Aspect	Accidental (undefined)	Deliberate (designed)
What happens on failure	Whatever the unhandled exception does	A specific, pre-decided fallback behavior
User experience	Blank section, spinner that never resolves, or a crash	Feature clearly absent, or replaced by a sensible default
Verified before it happens?	No — first observed during a real incident	Yes — tested by deliberately failing the dependency beforehand
Who decided the behavior	Nobody — it emerged from the code as written	An explicit design decision, documented per dependency

Remember: A degraded mode is a specific, designed, and tested behavior for "this dependency is down" — not an emergent accident of unhandled errors. Classifying a dependency as optional is a necessary first step but changes nothing on its own; the fallback code has to actually be written, and then deliberately exercised (not just theoretically covered by a try/except) to confirm it produces a genuinely usable result.

The four patterns

Stale cache, disabled feature, queued work, partial results — and matching each to the situation it actually fits.

Four patterns of graceful degradation

coreintermediate

Four recurring patterns cover most real graceful-degradation designs, each trading a specific piece of freshness or completeness for continued availability. Serving stale cache means that when the source of truth for some piece of data cannot be reached, the system serves the last known-good value instead of failing — the data might be a few minutes or hours old, but a slightly outdated price or profile picture is usually far less damaging than the whole page failing to load. Disabling a non-critical feature outright (the recommendations example used throughout this section) means the feature simply does not appear when its dependency is unavailable, rather than blocking or breaking the surrounding page. Queuing notifications (or any other write that does not need to happen synchronously) means the work is durably recorded and delivered once its dependency recovers, rather than being attempted and failed at the moment of the original request — the user's action still succeeds, and the notification arrives a little late instead of not at all. Returning partial results means that when a request depends on several independent sources and only some of them respond in time, the system returns what it has rather than waiting for or failing on the slowest or unavailable source — a search results page missing one source's results is a far better outcome than a page that shows nothing because it waited for every source to finish.

Think of it as

A restaurant that runs out of one ingredient does not close for the night — it takes that one dish off the specials board (disable a feature), serves yesterday's baked bread if today's is not out of the oven yet rather than serving no bread (stale cache), takes a phone order and calls the customer back once the kitchen catches up rather than making them wait on hold (queue the work), and, if a table ordered four dishes and one is delayed, serves the three that are ready rather than making the whole table wait for all four to arrive together (partial results). None of these responses pretend nothing is wrong — the customer can tell the bread is not fresh, or that one dish is missing — but every one of them keeps the restaurant open and the meal broadly successful, instead of turning one ingredient shortage into a fully cancelled dinner.

python

# The four patterns applied to one request:
# a dashboard aggregating four independent widgets

def get_pricing():
    try:
        return pricing_service.get_live_price(sku)
    except ServiceError:
        return cache.get_stale(f"price:{sku}")  # stale cache

def get_recommendations():
    try:
        return recommendations_service.get_upsells(sku)
    except ServiceError:
        return None  # disable: caller omits the section

def notify_price_drop(user, sku):
    queue.enqueue('send_price_drop_email', user, sku)
    # queued: succeeds immediately regardless of the
    # email service's current availability

def get_dashboard(widgets):
    results = {}
    for widget in widgets:
        try:
            results[widget.id] = widget.fetch(timeout=0.3)
        except (Timeout, ServiceError):
            continue  # partial results: skip, don't block
    return results

What we're doing: Trace a dashboard request that hits all four patterns at once when several widgets' dependencies are degraded.

dashboard-request-trace.txttext

Request: GET /dashboard for a logged-in user

pricing widget:          pricing-service times out
                          -> served last cached price
                             (2 minutes old)
recommendations widget:   recommendations-service down
                          -> section omitted entirely
notifications sent:       "price dropped" email
                          -> enqueued for async delivery,
                             not sent inline with this
                             request
overall dashboard:        4 of 5 widgets responded within
                          300ms; the 5th (weather widget)
                          did not -> dashboard renders
                          with 4 widgets, weather omitted

3: The price shown is 2 minutes stale rather than live — an acceptable trade for this use case (browsing a dashboard), which would not be acceptable on a page confirming an active price for checkout.
10: The email is queued rather than sent synchronously, so the dashboard request itself does not depend on the email service being available at all — the notification is delayed, not lost.
14: The dashboard as a whole applies the partial-results pattern at the top level: 4 working widgets are shown immediately rather than the whole page waiting on or failing because of the one slow widget.

Why this works: None of these four outcomes individually looks impressive — a stale price, a missing section, a delayed email, one missing widget — but together they add up to a dashboard that loaded successfully and usefully despite three simultaneous partial dependency problems, which is the entire value proposition of choosing the right degradation pattern per situation rather than one blanket "fail if anything is wrong" behavior.

Serving stale cache for data where staleness is actually dangerous

Wrong

python

def get_account_balance(account_id):
    try:
        return ledger_service.get_live_balance(account_id)
    except ServiceError:
        return cache.get_stale(f"balance:{account_id}")
        # used during an active funds transfer flow

Better

python

def get_account_balance(account_id):
    try:
        return ledger_service.get_live_balance(account_id)
    except ServiceError:
        raise BalanceUnavailable(
            "cannot confirm current balance right now")
        # transfer flow blocks rather than risking a
        # transfer approved against a stale balance

What you see: A user transfers funds based on a balance that was actually already spent moments earlier by a different, concurrent transaction, because the balance shown during the transfer flow was served from a two-minute-old cache rather than a live read.

Why: Stale cache is the right pattern exactly when staleness is an acceptable cost — a dashboard glance at a price is one such case, but a balance used to authorize a financial transfer is a case where correctness, not availability, is the requirement that must not be traded away, and applying the same pattern everywhere ignores that distinction.

Four degradation patterns

Stale cache

serve the last known-good value

Disable feature

hide it rather than show it broken

Queue the work

succeed now, deliver later

Partial results

return what responded in time

Four patterns and the situation each one fits

Four patterns and the situation each one fits
Pattern	What is traded away	Fits when
Serve stale cache	Freshness (data may be minutes/hours old)	A slightly outdated value beats no value, and staleness is acceptable
Disable the feature	The feature itself, entirely, for now	No reasonable substitute exists; hiding is clearer than showing broken output
Queue the work	Immediacy of a downstream effect	The work does not need to happen synchronously with the request
Return partial results	Completeness of the aggregate response	Some correct results beat none, and the caller can use a partial answer

Remember: Four patterns cover most graceful-degradation needs: serve stale cache (trade freshness for availability), disable the feature outright (when no reasonable substitute exists), queue the work (for anything that does not need a synchronous response), and return partial results (when some correct data beats none). Match the pattern to what can actually be traded away safely — staleness is fine for a dashboard price and dangerous for an active balance check — and make a partial result visibly partial, not silently incomplete.