beginnerintermediateadvanced

Python · Testing

Test types

Concepts: 2

The test pyramid — unit, integration, API, and end-to-end tests — plus three more specialized types: regression, contract, and performance tests.

This section

Python Tutorial: Unit Testing Your Code with the unittest ModuleCorey Schafer

The test pyramid

What each layer actually checks, and the speed/confidence tradeoff that decides how many of each to write.

The test pyramid: unit, integration, API, and end-to-end tests

coreintermediate

A unit test checks one function in isolation. An integration test checks two real pieces together. An API test checks a running service through HTTP. An end-to-end test drives the whole system like a real user.

Think of it as

Picture a pyramid: many fast, narrow unit tests at the base, fewer integration tests above them, and a handful of slow, broad end-to-end tests at the top. Each layer up costs more time and tells you less about exactly what broke, but proves more about whether the real system actually works.

python

# Roughly: many unit tests, fewer integration, fewer still API/E2E
def test_unit_example(): ...           # isolated, mocked dependencies
def test_integration_example(db): ...  # real component(s), no mocks
def test_api_example(client): ...      # real HTTP call to the service
def test_e2e_example(browser): ...     # drives the whole running system

What we're doing: Show the same behavior tested at two different layers — a unit test with a mocked dependency, and an integration test against a real one — to make the speed/confidence tradeoff concrete.

test_layers.pypython

from unittest.mock import Mock

def charge_customer(payment_gateway, amount):
    if amount <= 0:
        raise ValueError("amount must be positive")
    return payment_gateway.charge(amount)


def test_charge_customer_rejects_non_positive_amount():
    # UNIT: no real gateway needed -- this logic doesn't touch it
    fake_gateway = Mock()
    try:
        charge_customer(fake_gateway, -10)
        assert False, "expected ValueError"
    except ValueError as e:
        assert "positive" in str(e)


def test_charge_customer_calls_gateway_with_amount():
    # still a UNIT test -- gateway is mocked, not a real payment call
    fake_gateway = Mock(charge=Mock(return_value="charged"))
    result = charge_customer(fake_gateway, 50)
    assert result == "charged"
    fake_gateway.charge.assert_called_once_with(50)

10: This test only exercises the validation branch — a real payment gateway is irrelevant to it, so mocking is correct here.
17: Still a unit test: it checks that charge_customer calls the gateway correctly, without needing a real payment network.

Output

2 passed

Why this works: Both tests stay fast and isolated because the payment gateway — the expensive, external dependency — is mocked. An integration test would swap Mock() for a real (sandboxed) gateway client to prove the actual wiring works, at the cost of needing network access and running slower.

Writing only end-to-end tests because they feel most "real"

Wrong

python

# ONE giant end-to-end test covering login -> add to cart -> checkout
def test_full_purchase_flow(browser):
    browser.login("user@example.com", "password")
    browser.add_to_cart("widget")
    browser.checkout()
    assert browser.page_contains("Order confirmed")
    # if this fails, WHICH of the three steps broke?

Better

python

def test_login_succeeds_with_valid_credentials(): ...
def test_add_to_cart_increments_item_count(): ...
def test_checkout_charges_correct_total(): ...
# plus ONE e2e test for the full flow, not the only test

What you see: A single failing end-to-end test gives almost no clue which step broke, and the whole suite runs slowly because every test drives the full stack.

Why: End-to-end tests prove the pieces are wired together, but they are the wrong layer for pinpointing a specific bug — a broad failure with many possible causes is expensive to debug. Most coverage should live in fast, narrow unit tests; end-to-end tests confirm the pieces fit, in smaller numbers.

The pyramid — narrow and fast at the base, broad and slow at the top

End-to-end

the whole system, like a real user — slowest, broadest

API

a running service via HTTP — seconds

Integration

two+ real components together — seconds, real I/O

Unit

one function, dependencies mocked — milliseconds, most coverage lives here

The four everyday test types

The four everyday test types
Type	Checks	Speed
Unit	One function/class, dependencies mocked	Milliseconds — hundreds run in seconds
Integration	Two+ real components together (code + real DB)	Seconds — real I/O involved
API	A running service via its HTTP interface	Seconds — a real request/response cycle
End-to-end	The whole system, like a real user	Slowest — minutes, full stack involved

Together

python

# Unit -- no real database, no real network
def test_calculate_discount():
    assert calculate_discount(price=100, percent=10) == 90

# Integration -- a real (test) database
def test_save_order_persists_to_db(db_session):
    order = save_order(db_session, {"item": "widget", "qty": 2})
    assert db_session.query(Order).get(order.id) is not None

# API -- a real HTTP request against a running test server
def test_create_order_endpoint(client):
    response = client.post("/orders", json={"item": "widget", "qty": 2})
    assert response.status_code == 201

Remember: Unit tests are fast and narrow; integration/API/end-to-end tests grow slower and broader — build a pyramid.

Regression, contract, and performance tests

standardintermediate

A regression test locks in a specific bug fix so it cannot silently come back. A contract test checks that an API still matches what its consumers expect. A performance test checks that code stays inside a speed or resource budget.

Think of it as

A regression test is a tripwire planted exactly where something broke before — it exists because that specific bug happened once. A contract test is a tripwire on an API's shape, planted for consumers you may never see directly.

python

def test_regression_issue_1234_negative_quantity_no_longer_crashes():
    # Bug #1234: negative quantity raised an unhandled IndexError
    result = calculate_total(quantity=-1, price=10)
    assert result == 0   # now clamped, does not crash

def test_user_response_contract():
    response = get_user(42)
    assert set(response.keys()) == {"id", "email", "created_at"}

def test_search_completes_within_budget():
    import time
    start = time.perf_counter()
    search("widget")
    assert time.perf_counter() - start < 0.5

What we're doing: Write a regression test that locks in a real bug fix, and show it fails against the old (buggy) version.

test_regression.pypython

def calculate_total(quantity, price):
    # Bug #1234 fix: negative quantity used to raise an unhandled error.
    # Clamp to zero instead of letting it produce a nonsensical total.
    if quantity < 0:
        return 0
    return quantity * price


def test_regression_1234_negative_quantity_returns_zero():
    # Guards specifically against Bug #1234 coming back
    assert calculate_total(quantity=-1, price=10) == 0


def test_calculate_total_normal_case():
    assert calculate_total(quantity=3, price=10) == 30

2: The comment above the fix names the bug — this is what makes a regression test findable when someone later asks "why is this clamp here?"
9: The test name cites the bug number, not just the behavior — a future reader can trace it back to the original report.

Output

2 passed

Why this works: Removing the quantity < 0 clamp would make this specific test fail immediately, exactly where the original bug lived — that is the point of a regression test: it exists to catch one specific class of reintroduction, not to be a general-purpose check.

Remember: Regression tests guard a fixed bug from returning; contract tests guard an API's shape; performance tests guard a speed/resource budget.

See also: test pyramid and types · coverage