beginnerintermediateadvanced

Python · Test types and quality

Test types and test quality

The test pyramid, plus what actually makes a suite trustworthy — coverage, isolation, determinism, and testing failure paths

Unit → Integration → API → E2Efast and narrow at the base, slow and broad at the top
Coverageshows what never ran — not proof of correctness
Isolated + deterministicno shared state, no dependence on the real clock
Test failure paths toopytest.raises + mock.side_effect exercise error branches

1.Coverage

Coverage measures which lines of code actually ran during the test suite, as a percentage. It tells you what was NOT exercised at all — it does not tell you whether the lines that did run were tested correctly.

Key features

pytest --cov=mymodule reports the percentage of its lines executed by the test suite
--cov-report=term-missing shows exactly which line numbers were never run
High coverage does not mean correct — a line can run without its result being asserted on
100% coverage is rarely useful — often means testing trivial code just to hit the number
Coverage is most useful for finding UNTESTED code, not as a proxy for test quality

What coverage tells you, and what it does not

83% covered

line 7 never ran — a real, findable gap

100% covered, no assert

the line ran, but the result was never checked

Chasing the number

testing trivial getters instead of real edge cases

Command

python

# pytest --cov=mymodule --cov-report=term-missing
#
# Name          Stmts   Miss  Cover   Missing
# ---------------------------------------------
# mymodule.py       6      1    83%   8

Coverage shows which lines never ran — a floor for thoroughness, not proof of correctness.

2.Test isolation and determinism

An isolated test does not depend on any other test having run first, or on run order. A deterministic test gives the exact same pass/fail result every time — no reliance on the real clock, randomness, or network timing.

Key features

Isolated: does not depend on shared mutable state left behind by another test
Deterministic: same result every run — no dependence on real time, randomness, or order
Module/session-scoped fixtures returning mutable objects commonly cause shared state
datetime.now(), random.random(), and network calls are common non-determinism sources
A flaky test (passes then fails, no code change) is almost always an isolation problem

Order-dependent vs. isolated tests, same suite

shared_cart = [] (module-level)

test_add_item_to_cart appends, never resets
test_cart_starts_empty depends on run order
Passes alone, fails as part of the full file

now=datetime(2026,1,1) (explicit input)

No dependence on the real clock or randomness
Same result every run, any order, alone or together
Time and randomness become ordinary, controllable inputs

Command

python

# Non-deterministic -- depends on the real clock
def test_is_expired():
    order = Order(created_at=datetime.now())
    assert not order.is_expired()

# Deterministic -- the "clock" is an explicit input
def test_is_expired():
    order = Order(created_at=datetime(2026, 1, 1))
    assert order.is_expired(now=datetime(2026, 2, 1))

An isolated test ignores other tests' side effects; a deterministic test ignores the real clock and randomness.

3.Database test isolation and external service testing

A database test needs a strategy to stay isolated — usually a transaction rolled back after each test. An external service is usually faked in most tests, with a small number of real, network-hitting tests kept separate and marked.

Key features

A per-test transaction rolled back afterward is the cheapest way to isolate database tests
A fixture commonly wraps setup: begin a transaction, yield the session, roll it back after
External services are usually faked in most tests, real calls kept to a small marked set
@pytest.mark.integration lets a suite exclude network tests from the fast, default run
A fake in-process test double for a database trades realism for speed in unit tests

Command

python

@pytest.fixture
def db_session():
    connection = engine.connect()
    transaction = connection.begin()
    session = Session(bind=connection)
    yield session
    session.close()
    transaction.rollback()   # undoes every write this test made
    connection.close()

@pytest.mark.integration    # excluded from the fast default run
def test_real_payment_api():
    response = real_payment_client.charge(...)
    assert response.status == "succeeded"

Isolate database tests with a transaction rollback; fake external services, marking the few real ones.

4.Testing failure scenarios

A test suite that only checks correct inputs misses how code behaves when things go wrong. pytest.raises(...) asserts that an error actually happens, and a mocked dependency can be told to fail on demand.

Key features

with pytest.raises(ExceptionType): asserts the block raises that exact exception type
pytest.raises(ExceptionType, match="text") also checks the message against a regex
A mocked dependency can be told to raise: mock.side_effect = ConnectionError("timeout")
Worth testing: empty input, timeout, malformed data, permission denied, resource exhausted
A suite with 100% happy-path coverage can still crash on the first real production error

Deliberately exercising the failure branch

mock.side_effect = ConnectionError

forces the dependency to fail on demand

fetch_with_fallback(mock, url)

the except branch actually runs

pytest.raises(ValueError, match=...)

asserts the RIGHT error, not just any error

Command

python

def test_raises_on_invalid_input():
    with pytest.raises(ValueError, match="must be positive"):
        process_amount(-5)

def test_handles_network_failure(mock_client):
    mock_client.get.side_effect = ConnectionError("timeout")
    result = fetch_with_retry(mock_client, url)
    assert result is None   # fails gracefully, does not crash

pytest.raises asserts an error happens; mock.side_effect forces a dependency to fail on demand.

Spotted a mistake on this page? Tell us.

Read Sheet