Filter concepts by levelShowing all levels.

Python · Testing

Quality

Concepts
4

What makes a test suite trustworthy beyond "the tests pass" — coverage, isolation, determinism, database/external-service test isolation, and deliberately testing failure scenarios.

This section

Trustworthy test suites

Measuring what ran, keeping tests independent and repeatable, isolating database writes, and testing the failure paths as deliberately as the success ones.

Coverage

coreintermediate

Coverage measures which lines of code actually ran during the test suite, as a percentage. It tells you what was NOT exercised at all — it does not tell you whether the lines that did run were tested correctly.

Think of it as

Coverage is a checklist of which rooms in a house someone walked through, not whether they checked anything in each room. 100% coverage means every room was visited — it says nothing about whether the visitor actually looked in the closets.

python
# pytest --cov=mymodule --cov-report=term-missing
#
# Name          Stmts   Miss  Cover   Missing
# ---------------------------------------------
# mymodule.py       6      1    83%   8

What we're doing: Run coverage against a module with an untested branch and see exactly which line it flags as missing.

calc.py + test_calc.pypython
# calc.py
def add(a, b):
    return a + b

def divide(a, b):
    if b == 0:
        raise ValueError("cannot divide by zero")
    return a / b

# test_calc.py
from calc import add, divide
import pytest

def test_add():
    assert add(2, 3) == 5

def test_divide_by_zero_raises():
    with pytest.raises(ValueError, match="cannot divide by zero"):
        divide(1, 0)
7
This return line is never reached — every test either adds, or divides by zero and hits the raise on the line above.
Output
Name      Stmts   Miss  Cover   Missing
---------------------------------------
calc.py       6      1    83%   8
---------------------------------------
TOTAL         6      1    83%

Why this works: Coverage counts calc.py at 6 executable statements; 5 ran, 1 (the successful division's return a / b) never did, because no test calls divide with a nonzero denominator. --cov-report=term-missing names the exact line so it is easy to find and fix.

Treating a high coverage number as proof the code is well tested

Wrong

python
def calculate_discount(price, percent):
    return price - (price * percent / 100)

def test_calculate_discount():
    calculate_discount(100, 10)   # runs the line -- 100% coverage
    # no assert at all -- the RESULT is never checked

Better

python
def test_calculate_discount():
    result = calculate_discount(100, 10)
    assert result == 90   # actually checks the computed value

What you see: Coverage reports 100%, but the test suite would not catch calculate_discount returning the wrong value — a broken implementation still passes.

Why: Coverage only measures whether a line EXECUTED, never whether its result was correct. A test that calls code without asserting on the outcome inflates the coverage number without testing anything real.

What coverage tells you, and what it does not

83% covered

line 7 never ran — a real, findable gap

100% covered, no assert

the line ran, but the result was never checked

Chasing the number

testing trivial getters instead of real edge cases

  • 83% covered — line 7 never ran — a real, findable gap
  • 100% covered, no assert — the line ran, but the result was never checked
  • Chasing the number — testing trivial getters instead of real edge cases

Remember: Coverage shows which lines never ran — a floor for thoroughness, not proof of correctness.

See also: test pyramid and types · testing failure scenarios

Test isolation and determinism

coreintermediate

An isolated test does not depend on any other test having run first, or on run order. A deterministic test gives the exact same pass/fail result every time — no reliance on the real clock, randomness, or network timing.

Think of it as

Every test should be able to run alone, in any order, any number of times, and get the same answer. If a test only passes when run after another specific test, it is not testing its own behavior — it is testing an accident of execution order.

python
# Non-deterministic -- depends on the real clock
def test_is_expired():
    order = Order(created_at=datetime.now())
    assert not order.is_expired()

# Deterministic -- the "clock" is an explicit input
def test_is_expired():
    order = Order(created_at=datetime(2026, 1, 1))
    assert order.is_expired(now=datetime(2026, 2, 1))

What we're doing: Show a test suite where module-level shared state makes one test's result depend on whether another test ran first — a real isolation failure.

test_isolation.pypython
shared_cart = []   # module-level state -- BAD: shared across tests

def test_add_item_to_cart():
    shared_cart.append("widget")
    assert len(shared_cart) == 1

def test_cart_starts_empty():
    # depends entirely on whether test_add_item_to_cart already ran
    assert len(shared_cart) == 0
1
A module-level list is shared by every test in the file — nothing resets it between tests.
9
This assertion's outcome depends on test order, not on anything test_cart_starts_empty itself does — that is the isolation failure.
Output
test_add_item_to_cart PASSED
test_cart_starts_empty FAILED
E       assert 1 == 0

Why this works: shared_cart is created once when the module loads and never reset — test_add_item_to_cart's side effect leaks into test_cart_starts_empty. Running test_cart_starts_empty ALONE would pass; running the full file fails, which is the exact symptom of a real isolation bug.

A test that depends on today's real date

Wrong

python
from datetime import datetime, timedelta

def test_subscription_not_expired():
    expires = datetime.now() + timedelta(days=1)
    assert not is_expired(expires)
    # passes today -- but is_expired's own logic is never really isolated
    # from the ACTUAL current time when the test runs

Better

python
from datetime import datetime

def test_subscription_not_expired():
    now = datetime(2026, 1, 1)
    expires = datetime(2026, 1, 2)
    assert not is_expired(expires, now=now)
    # deterministic: no dependence on when the test actually runs

What you see: A test that passed for months suddenly starts failing with no code change — often around a date boundary (midnight, month-end, a leap year) the test never accounted for.

Why: Any code that reads the real clock inside a test makes the test's outcome depend on when it happens to run. Passing "now" explicitly makes time an ordinary, controllable input instead of a hidden global that changes every second.

Order-dependent vs. isolated tests, same suite

shared_cart = [] (module-level)

  • +test_add_item_to_cart appends, never resets
  • +test_cart_starts_empty depends on run order
  • +Passes alone, fails as part of the full file

now=datetime(2026,1,1) (explicit input)

  • No dependence on the real clock or randomness
  • Same result every run, any order, alone or together
  • Time and randomness become ordinary, controllable inputs
  • shared_cart = [] (module-level)
    • test_add_item_to_cart appends, never resets
    • test_cart_starts_empty depends on run order
    • Passes alone, fails as part of the full file
  • now=datetime(2026,1,1) (explicit input)
    • No dependence on the real clock or randomness
    • Same result every run, any order, alone or together
    • Time and randomness become ordinary, controllable inputs

Remember: An isolated test ignores other tests' side effects; a deterministic test ignores the real clock and randomness.

See also: fixtures and scopes · database and external service testing

Database test isolation and external service testing

standardintermediate

A database test needs a strategy to stay isolated — usually a transaction rolled back after each test. An external service is usually faked in most tests, with a small number of real, network-hitting tests kept separate and marked.

Think of it as

Database isolation is giving each test its own sandbox that gets swept clean afterward — a transaction that never actually commits is the cheapest version of that sandbox. External services are usually simulated entirely, because a real API you do not control can be slow, rate-limited, or simply down.

python
@pytest.fixture
def db_session():
    connection = engine.connect()
    transaction = connection.begin()
    session = Session(bind=connection)
    yield session
    session.close()
    transaction.rollback()   # undoes every write this test made
    connection.close()

@pytest.mark.integration    # excluded from the fast default run
def test_real_payment_api():
    response = real_payment_client.charge(...)
    assert response.status == "succeeded"

What we're doing: Show a fixture pattern that isolates database writes with a rollback, keeping every test's changes invisible to every other test.

db_isolation_pattern.pypython
import pytest

class FakeSession:
    """Stands in for a real DB session -- rollback just clears in-memory state."""
    def __init__(self):
        self._rows = []

    def add(self, row):
        self._rows.append(row)

    def rollback(self):
        self._rows.clear()   # simulates the real transaction rollback


@pytest.fixture
def db_session():
    session = FakeSession()
    yield session
    session.rollback()   # runs after the test -- every write disappears


def test_insert_visible_within_test(db_session):
    db_session.add({"id": 1, "name": "widget"})
    assert len(db_session._rows) == 1
3
This fake stands in for a real ORM session so the pattern can be demonstrated without a real database connection.
15
The fixture itself calls rollback() AFTER the test finishes (after yield) — this is where isolation actually happens, not inside the test.
22
The test only sees the write it made — the next test gets a brand new FakeSession, with the previous rollback already applied.
Output
1 passed

Why this works: The pattern matters more than this specific fake: a real db_session fixture wraps each test in a transaction that never commits, so no test's writes are ever visible to the next one — the same guarantee this simplified example demonstrates directly.

Remember: Isolate database tests with a transaction rollback; fake external services, marking the few real ones.

See also: test isolation and determinism · test doubles vocabulary · marks and plugins

Testing failure scenarios

coreintermediate

A test suite that only checks correct inputs misses how code behaves when things go wrong. pytest.raises(...) asserts that an error actually happens, and a mocked dependency can be told to fail on demand.

Think of it as

Happy-path tests prove the code works when everything goes right — which is the easy case. Failure-scenario tests prove the code fails SAFELY when something goes wrong, which is usually the more important guarantee in production.

python
def test_raises_on_invalid_input():
    with pytest.raises(ValueError, match="must be positive"):
        process_amount(-5)

def test_handles_network_failure(mock_client):
    mock_client.get.side_effect = ConnectionError("timeout")
    result = fetch_with_retry(mock_client, url)
    assert result is None   # fails gracefully, does not crash

What we're doing: Test both that an error is correctly raised, and that a mocked dependency failing does not crash the calling code.

test_failure_scenarios.pypython
import pytest
from unittest.mock import Mock

def validate_amount(amount):
    if amount <= 0:
        raise ValueError("amount must be positive")
    return amount


def test_validate_amount_raises_on_negative():
    with pytest.raises(ValueError, match="must be positive"):
        validate_amount(-10)


def fetch_with_fallback(client, url):
    try:
        return client.get(url)
    except ConnectionError:
        return None   # fails gracefully instead of crashing the caller


def test_fetch_with_fallback_handles_connection_error():
    mock_client = Mock()
    mock_client.get.side_effect = ConnectionError("timeout")
    result = fetch_with_fallback(mock_client, "https://api.example.com")
    assert result is None
10
pytest.raises confirms the ValueError actually happens, with the expected message — not just that SOME error occurs.
23
side_effect makes the mock raise ConnectionError when called, simulating a real network failure without a real network.
Output
2 passed

Why this works: pytest.raises turns "this should fail" into a real, checked assertion instead of an untested assumption. side_effect lets a test simulate exactly the kind of failure — a timeout, a malformed response — that a happy-path test would never trigger on its own.

Only testing that code works, never that it fails correctly

Wrong

python
def test_fetch_data():
    client = RealClient()
    result = fetch_with_fallback(client, "https://api.example.com")
    assert result is not None
    # what happens when the network call actually fails? never tested.

Better

python
def test_fetch_data_success(mock_client):
    mock_client.get.return_value = {"data": "ok"}
    assert fetch_with_fallback(mock_client, url) == {"data": "ok"}

def test_fetch_data_handles_failure(mock_client):
    mock_client.get.side_effect = ConnectionError("timeout")
    assert fetch_with_fallback(mock_client, url) is None

What you see: Code that is supposed to "fail gracefully" instead crashes in production the first time the network actually times out, because no test ever exercised that path.

Why: A happy-path-only suite proves the success case works but says nothing about the failure branch — side_effect on a mock is the deliberate way to force that branch to run under test, instead of hoping it works.

Deliberately exercising the failure branch

mock.side_effect = ConnectionError

forces the dependency to fail on demand

fetch_with_fallback(mock, url)

the except branch actually runs

pytest.raises(ValueError, match=...)

asserts the RIGHT error, not just any error

  1. mock.side_effect = ConnectionError — forces the dependency to fail on demand
  2. fetch_with_fallback(mock, url) — the except branch actually runs
  3. pytest.raises(ValueError, match=...) — asserts the RIGHT error, not just any error

Remember: pytest.raises asserts an error happens; mock.side_effect forces a dependency to fail on demand.

See also: coverage · what to mock · try except

Advertisement