Filter concepts by levelShowing all levels.

Django · Section 27

PostgreSQL with Django

Level
advanced
Read
30 min
Concepts
6

PostgreSQL knowledge a Django engineer needs independently of the ORM, per the roadmap's own framing. Tables/rows/columns, primary/foreign keys, unique/check constraints, joins, CTEs, and window functions are already covered by the standalone PostgreSQL topic (dedup'd here, not re-derived). The five genuinely new concepts: index types and when each earns its place over the B-tree default; transaction isolation levels and what each actually guarantees; locks and the deadlock detect-and-abort mechanism, plus the real prevention (consistent lock ordering); reading EXPLAIN/EXPLAIN ANALYZE query plans to spot a missing index or stale statistics; VACUUM/ANALYZE and why MVCC makes them necessary; and connection pooling, replication lag, and table partitioning as the three answers to outgrowing a single instance/table.

This section

What is true here

  1. B-tree is the default and handles equality/range on ordinary columns; GIN indexes arrays/JSONB/full-text containment; GiST handles geometric/nearest-neighbor; BRIN is a tiny index for huge, insertion-ordered tables.
  2. Read Committed (PostgreSQL's default) re-establishes a snapshot per statement, not once per transaction — Repeatable Read and Serializable snapshot once and both require the application to catch and retry a serialization failure.
  3. A deadlock is a circular wait between transactions — PostgreSQL detects and breaks it by aborting one, unpredictably; the real defense is always acquiring locks on multiple rows/tables in a consistent order.
  4. EXPLAIN shows the planner's estimate without running anything; EXPLAIN ANALYZE actually executes the query — a large estimate-vs-actual gap usually signals stale statistics, fixable with ANALYZE.
  5. MVCC means UPDATE/DELETE leave old row versions behind — VACUUM reclaims that space and prevents transaction-id wraparound; a connection pooler (not a higher max_connections) is the standard fix for too many concurrent connections; async replication can lag, so read-your-own-write queries should stay on the primary.

What you will be able to do

  • Choose the right PostgreSQL index type for a given query shape
  • Reason correctly about what each transaction isolation level actually guarantees
  • Prevent deadlocks via consistent lock ordering rather than relying on detection alone
  • Read an EXPLAIN ANALYZE plan to find a missing index or stale statistics
  • Explain why VACUUM/ANALYZE are necessary under MVCC, and when connection pooling/partitioning become the right tool

Indexing and reading query plans

Choosing the right index type, and reading EXPLAIN/EXPLAIN ANALYZE to confirm it actually helped.

Index types

coreintermediate

B-tree is the default and correct choice for most cases — equality and range queries (<, <=, =, >=, >, BETWEEN, IN) on sortable data. GIN indexes arrays/JSONB/full-text (containment queries: does this array/document contain X). GiST handles geometric/nearest-neighbor queries. BRIN is a tiny, cheap index for huge tables where a column's values roughly follow physical row order (e.g. an auto-incrementing id or a timestamp).

Think of it as

An index is a trade: faster reads for a specific access pattern, at the cost of extra storage and slower writes (every INSERT/UPDATE has to update every index on that table too). B-tree earns its status as the default because most real queries are equality/range lookups on ordinary columns — everything else (GIN, GiST, BRIN) exists because SOME data shape breaks B-tree's assumptions. A JSONB column asking "does this document contain key X" isn't a range query at all — GIN is built for exactly that shape. BRIN takes the opposite bet from B-tree: instead of indexing every row precisely, it stores just a min/max per block, betting that a huge table's values are already roughly sorted by insertion order — tiny index, slightly less precise, but the only realistic option once a table gets large enough that a full B-tree index itself becomes expensive to maintain.

sql
CREATE INDEX idx_name ON table_name (column_name);            -- B-tree
CREATE INDEX idx_name ON table_name USING GIN (jsonb_column);  -- GIN

What we're doing: Index a JSONB column for fast "contains key" lookups, and confirm a plain B-tree index would not help this query shape.

migration.sqlsql
CREATE INDEX idx_products_attributes ON products USING GIN (attributes);

-- now fast:
SELECT * FROM products WHERE attributes @> '{"color": "red"}';
1
USING GIN is required here — a default B-tree index on a JSONB column can only accelerate exact whole-column equality, not "does this document contain this key/value," which is what @> (containment) actually needs.

Why this works: A B-tree index on attributes would only speed up queries comparing the ENTIRE JSONB value for equality — the containment query (@>, "does this document have color=red among possibly many other keys") needs an index that understands the document's internal structure, which is exactly what GIN provides for JSONB/array columns.

Adding a B-tree index on a JSONB column and expecting containment queries to speed up

Wrong

sql
CREATE INDEX idx_products_attributes ON products (attributes);  -- default B-tree
SELECT * FROM products WHERE attributes @> '{"color": "red"}';   -- still a sequential scan

Better

sql
CREATE INDEX idx_products_attributes ON products USING GIN (attributes);
SELECT * FROM products WHERE attributes @> '{"color": "red"}';   -- now uses the GIN index

What you see: EXPLAIN on the containment query still shows a Seq Scan even after adding an index — the index exists but the query planner never uses it for this operator.

Why: A B-tree index only supports the operators B-tree knows how to order (<, <=, =, >=, >) — @> (containment) is not one of them, so the planner correctly ignores a B-tree index for this query and falls back to scanning every row. GIN is built specifically to index composite values by their contents, which is what containment operators need.

Choosing an index type by query shape

B-tree (default)

equality/range on ordinary columns

GIN

arrays/JSONB/full-text containment

GiST

geometric / nearest-neighbor

BRIN

huge, insertion-ordered tables

  1. B-tree (default) — equality/range on ordinary columns
  2. GIN — arrays/JSONB/full-text containment
  3. GiST — geometric / nearest-neighbor
  4. BRIN — huge, insertion-ordered tables

Choosing an index type

Choosing an index type
Query shapeIndex type
Equality/range on an ordinary columnB-tree (default)
"Does this array/JSONB contain X" / full-text searchGIN
Geometric / nearest-neighborGiST
Huge table, values roughly ordered by insertionBRIN

Together

sql
CREATE INDEX idx_orders_customer ON orders (customer_id);              -- B-tree, default
CREATE INDEX idx_orders_tags ON orders USING GIN (tags);                -- array/JSONB containment
CREATE INDEX idx_events_created_at ON events USING BRIN (created_at);   -- huge, insertion-ordered table

Remember: B-tree is the default and handles equality/range on ordinary columns; GIN is for arrays/JSONB/full-text containment queries; GiST is for geometric/nearest-neighbor; BRIN is a tiny index for huge, insertion-ordered tables. Every index has a real write cost — add them in response to an observed query pattern, not speculatively.

See also: query plans · vacuum and analyze · primary foreign unique not null check

Query plans: EXPLAIN and EXPLAIN ANALYZE

coreadvanced

EXPLAIN shows the query planner's ESTIMATED plan (node types, estimated cost, estimated row counts) without running the query. EXPLAIN ANALYZE actually RUNS the query and adds real measured numbers (actual time, actual rows, loop count) alongside the estimates — comparing the two is how a wildly-wrong estimate (usually from stale statistics) gets caught. A Seq Scan on a large table where an index exists, or "rows removed by filter" in the thousands, are the two most common red flags.

Think of it as

EXPLAIN alone is the planner's PREDICTION — built from table statistics, before anything runs. EXPLAIN ANALYZE is the REALITY CHECK — it actually executes the query and reports what really happened, side by side with what was predicted. A big gap between estimated and actual (either row counts or cost/time) is the signal something is wrong — usually stale statistics (fixed by ANALYZE), a missing index, or a query shape the planner can't reason about well. Reading a plan is fundamentally about finding the node doing far more work than the query's actual result size would suggest — a Seq Scan filtering 995,000 rows down to 5,000 matches is the plan quietly telling you an index on that column is missing.

sql
EXPLAIN SELECT ...;           -- estimated plan only, safe, doesn't execute
EXPLAIN ANALYZE SELECT ...;   -- actually runs the query, adds real timing

What we're doing: Diagnose a slow query with EXPLAIN ANALYZE, spot the missing-index signal, and confirm the fix with a second EXPLAIN ANALYZE.

diagnose.sqlsql
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123;
-- Seq Scan on orders (actual time=0.045..2500.000 rows=5000 loops=1)
--   Rows Removed by Filter: 995000

CREATE INDEX idx_orders_customer_id ON orders (customer_id);
ANALYZE orders;

EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123;
-- Index Scan using idx_orders_customer_id (actual time=0.051..125.000 rows=5000 loops=1)
1
995,000 rows removed by filter, for only 5,000 matches, is the plan directly reporting that 99.5% of the scanned work was wasted — the clearest possible signal that an index on customer_id would help.
4
ANALYZE after CREATE INDEX ensures the planner has fresh statistics reflecting the new index — without it, the planner might still choose the old Seq Scan plan out of habit until statistics catch up on their own.

Why this works: Guessing at performance fixes without EXPLAIN ANALYZE risks solving the wrong problem entirely — the "before" plan makes the actual bottleneck (a full scan of 1M rows) undeniable, and re-running EXPLAIN ANALYZE after the fix (actual time 2500ms → 125ms) is the only real confirmation the index actually helped, rather than assuming it did.

Reading only EXPLAIN (estimates), never EXPLAIN ANALYZE (actuals), when diagnosing a genuinely slow query

Wrong

sql
EXPLAIN SELECT * FROM orders WHERE customer_id = 123;
-- shows estimated cost=35000 — "looks expensive," but no real timing to confirm it's actually the slow part

Better

sql
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123;
-- actual time=0.045..2500.000 — confirms this query genuinely takes 2.5 real seconds

What you see: Time spent "optimizing" a query based on its estimated cost number, when the estimate was itself wrong (stale statistics) — the actual bottleneck in the application might be a completely different query that looked cheap in EXPLAIN but is called far more often, or has a much larger real/estimated gap.

Why: EXPLAIN's cost numbers are the planner's PREDICTION, built from potentially-stale statistics — they are useful for comparing two candidate query shapes relative to each other, but are not a reliable measure of real-world time on their own. EXPLAIN ANALYZE's actual time is the only number that reflects what genuinely happened when the query ran.

Reading a plan — the missing-index red flag

Seq Scan on orders (cost=0.00..35000.00 rows=5000 width=200) (actual time=0.045..2500.000 rows=5000 loops=1) Filter: (customer_id = 123) Rows Removed by Filter: 995000

Seq Scan on orders

Seq Scan — scans every row — a red flag on a large table with a selective WHERE

actual time=0.045..2500.000

actual time — from EXPLAIN ANALYZE — real measured time, not an estimate

Rows Removed by Filter: 995000

Rows Removed by Filter — 99.5% wasted scan — the clearest missing-index signal

  • Whole: Seq Scan on orders (cost=0.00..35000.00 rows=5000 width=200) (actual time=0.045..2500.000 rows=5000 loops=1) Filter: (customer_id = 123) Rows Removed by Filter: 995000
  • Seq Scan on orders — Seq Scan: scans every row — a red flag on a large table with a selective WHERE
  • actual time=0.045..2500.000 — actual time: from EXPLAIN ANALYZE — real measured time, not an estimate
  • Rows Removed by Filter: 995000 — Rows Removed by Filter: 99.5% wasted scan — the clearest missing-index signal

Common plan-reading red flags

Common plan-reading red flags
What you seeLikely cause
Seq Scan on a large table with a selective WHEREmissing index, or the planner isn't using an existing one
Large "Rows Removed by Filter"the filter should be index-backed instead of scan-and-discard
Estimated rows far off from actual rows (EXPLAIN ANALYZE)stale statistics — run ANALYZE
A node with a high loop countmultiply actual time × loops for the real total cost of that node

Together

sql
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123;
--  Seq Scan on orders (cost=0.00..35000.00 rows=5000 width=200)
--                      (actual time=0.045..2500.000 rows=5000 loops=1)
--    Filter: (customer_id = 123)
--    Rows Removed by Filter: 995000    <- red flag: scanned 1M rows for 5000 matches

Remember: EXPLAIN shows the planner's estimate without running anything; EXPLAIN ANALYZE actually executes the query and adds real timing — a big estimate-vs-actual gap usually means stale statistics (run ANALYZE). A Seq Scan with a large "Rows Removed by Filter" on a big table is the clearest missing-index signal. Wrap EXPLAIN ANALYZE on a modifying statement in BEGIN/ROLLBACK.

See also: indexes · vacuum and analyze · the n plus 1 pattern

Advertisement

Transactions, locking, and maintenance

Isolation levels, the deadlock detect-and-abort mechanism, and why VACUUM/ANALYZE exist under MVCC.

Transactions and isolation levels

coreintermediate

A transaction (BEGIN...COMMIT/ROLLBACK) groups statements so they succeed or fail together. PostgreSQL offers three isolation levels — Read Committed (the default: each statement sees data committed as of when THAT statement started), Repeatable Read (the whole transaction sees one consistent snapshot from its start), and Serializable (behaves as if transactions ran one at a time, detecting and rejecting conflicts). Stricter isolation prevents more anomalies but can raise a serialization error the application must be ready to retry.

Think of it as

Isolation level answers one question: "what happens when another transaction changes data WHILE mine is running?" Read Committed is the loosest, cheapest answer — each individual statement gets a fresh look at committed data, so two SELECTs in the same transaction can see different results if something else committed in between. Repeatable Read locks in a single consistent snapshot for the WHOLE transaction — no more surprises mid-transaction, but now two concurrent transactions trying to update the same row can conflict, and PostgreSQL makes one of them fail with a serialization error rather than silently letting them corrupt each other's work. Serializable is the strictest promise — the database guarantees the outcome is equivalent to SOME serial (one-at-a-time) ordering of all transactions, even if they actually ran concurrently, at the cost of applications needing to handle (and retry) serialization failures more often.

sql
BEGIN;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
-- statements here
COMMIT;

What we're doing: Use Serializable isolation for a transaction where a subtle read/write conflict with a concurrent transaction must never silently corrupt the result, and handle the resulting error case.

transfer.sqlsql
BEGIN;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
-- if COMMIT fails with "could not serialize access due to read/write dependencies",
-- the entire transaction must be retried from BEGIN
2
SERIALIZABLE is set explicitly — PostgreSQL's default (Read Committed) would allow this exact transfer to interleave with a concurrent one in a way that could produce a subtly wrong final balance under some access patterns.

Why this works: A balance transfer is exactly the case where "each statement sees whatever is currently committed" (Read Committed) is not always strong enough — Serializable guarantees the final result is equivalent to running every concurrent transaction one at a time, at the cost of the application needing to catch and retry the occasional serialization failure.

Setting Serializable/Repeatable Read isolation without ever handling the serialization-failure error

Wrong

python
with connection.cursor() as cursor:
    cursor.execute("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE")
    cursor.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")
    # no try/except — a serialization failure crashes the request entirely

Better

python
for attempt in range(3):
    try:
        with transaction.atomic():
            cursor.execute("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE")
            cursor.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")
        break
    except OperationalError as e:
        if "could not serialize access" not in str(e) or attempt == 2:
            raise

What you see: A request that should have simply retried and succeeded instead surfaces a raw database error (or a 500) to the end user, under normal, expected concurrent load — not a rare edge case, but the documented, intended behavior of this isolation level under contention.

Why: A serialization failure at Repeatable Read/Serializable is not a bug or an unusual event — PostgreSQL's own documentation states applications using these levels "must be prepared to retry transactions" that fail this way. Code that sets a stricter isolation level without a retry loop has only gotten half of what that isolation level is actually meant to provide.

Three isolation levels, stricter going up

Serializable

behaves as if transactions ran one at a time — retry on conflict

Repeatable Read

one consistent snapshot for the whole transaction

Read Committed (default)

each statement sees a fresh snapshot of committed data

  1. Serializable — behaves as if transactions ran one at a time — retry on conflict
  2. Repeatable Read — one consistent snapshot for the whole transaction
  3. Read Committed (default) — each statement sees a fresh snapshot of committed data

What each isolation level prevents

What each isolation level prevents
LevelPreventsStill possible
Read Committed (default)dirty readsnon-repeatable reads, phantom reads
Repeatable Readdirty reads, non-repeatable reads, phantom readsserialization anomalies (raises an error instead)
Serializableall of the abovenothing — strictest guarantee, retry required on conflict

Together

sql
BEGIN;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SELECT balance FROM accounts WHERE id = 1;   -- snapshot taken here
-- ... another transaction commits a change to this row ...
SELECT balance FROM accounts WHERE id = 1;   -- SAME value as before — the snapshot doesn't change
COMMIT;

Remember: Read Committed (default) re-snapshots per statement; Repeatable Read snapshots once for the whole transaction; Serializable adds full conflict detection. Repeatable Read and Serializable both require the application to catch and retry a serialization failure — that is expected behavior at those levels, not a bug.

See also: locks and deadlocks · atomic and nested blocks · recognizing the pattern

Locks and deadlocks

coreintermediate

A lock prevents two transactions from conflicting over the same row/table at once — UPDATE/DELETE/SELECT FOR UPDATE all take row-level locks automatically. A deadlock happens when two transactions each hold a lock the other one wants, so neither can proceed — PostgreSQL detects this automatically and aborts one of the transactions (unpredictable which one) so the other can continue. The real fix is prevention: always acquire locks on multiple objects in the same, consistent order across every code path.

Think of it as

Every UPDATE/DELETE implicitly locks the rows it touches until the transaction commits or rolls back — this is automatic and invisible most of the time, since most transactions touch different rows and never collide. A deadlock is the specific, circular case: Transaction A locks row 1 then wants row 2; Transaction B locks row 2 then wants row 1 — both wait forever unless something intervenes. PostgreSQL's deadlock detector is the safety net, not the fix — it breaks the cycle by killing one transaction, but WHICH one is essentially arbitrary, so the only real defense is making sure every code path that touches multiple rows/tables always acquires them in the same order, so the circular-wait shape can never form in the first place.

sql
-- always lock/update rows in the SAME order across every code path
UPDATE accounts SET balance = balance - 100 WHERE id = LEAST(1, 2);
UPDATE accounts SET balance = balance + 100 WHERE id = GREATEST(1, 2);

What we're doing: Prevent a deadlock in a funds-transfer function by always locking the two accounts in a fixed order, regardless of which account is the source and which is the destination.

accounts/services.pypython
def transfer(from_id, to_id, amount):
    first_id, second_id = sorted([from_id, to_id])
    with transaction.atomic():
        Account.objects.select_for_update().filter(id__in=[first_id, second_id]).order_by("id")
        Account.objects.filter(id=from_id).update(balance=F("balance") - amount)
        Account.objects.filter(id=to_id).update(balance=F("balance") + amount)
2
sorted([from_id, to_id]) guarantees the SAME two accounts are always locked in the same order, regardless of whether this specific call transfers 1→2 or 2→1.
3
Locking both accounts in that fixed order — before either update — means a concurrent transfer(2, 1, ...) call locks them in the identical order, so the two transactions can never form the circular-wait shape a deadlock requires.

Why this works: Without a fixed lock order, transfer(1, 2, 50) running concurrently with transfer(2, 1, 30) can lock account 1 then wait for account 2 in one transaction, while the other locks account 2 then waits for account 1 — the exact deadlock shape. Sorting the ids first ensures both transactions always approach the two accounts in the identical order, making the circular wait structurally impossible.

Locking rows in caller-determined order instead of a fixed, consistent order

Wrong

python
def transfer(from_id, to_id, amount):
    with transaction.atomic():
        Account.objects.select_for_update().get(id=from_id)   # order depends on caller's args
        Account.objects.select_for_update().get(id=to_id)
        # ...update both...

Better

python
def transfer(from_id, to_id, amount):
    first_id, second_id = sorted([from_id, to_id])
    with transaction.atomic():
        list(Account.objects.select_for_update().filter(id__in=[first_id, second_id]).order_by("id"))
        # ...update both...

What you see: Under real concurrent load — two transfers between the same pair of accounts, in opposite directions, happening around the same time — one of the two transactions occasionally fails with a deadlock error, intermittently and hard to reproduce on demand.

Why: Locking rows in whatever order the function's own parameters happen to list them means the lock order literally depends on which direction each caller's transfer runs — two opposite-direction transfers between the same two accounts lock them in opposite orders, which is precisely the circular-wait setup a deadlock requires. A fixed, data-derived order (e.g. sorting by id) removes that dependency entirely.

The classic circular-wait deadlock
Transaction A
row 1
row 2
Transaction B
  1. 1. locks row 1
  2. 2. locks row 2
  3. 3. wants row 2 — waits
  4. 4. wants row 1 — waits
  1. Transaction A → row 1: locks row 1
  2. Transaction B → row 2: locks row 2
  3. Transaction A → row 2: wants row 2 — waits
  4. Transaction B → row 1: wants row 1 — waits

Deadlock: the classic circular-wait shape

Deadlock: the classic circular-wait shape
StepTransaction ATransaction B
1locks row 1locks row 2
2wants row 2 — waitswants row 1 — waits
3stuckstuck — PostgreSQL detects the cycle and aborts one

Together

sql
-- Transaction A                    -- Transaction B
BEGIN;                                BEGIN;
UPDATE accounts SET ... WHERE id=1;   UPDATE accounts SET ... WHERE id=2;
UPDATE accounts SET ... WHERE id=2;   UPDATE accounts SET ... WHERE id=1;  -- DEADLOCK
-- one of these two transactions gets aborted by PostgreSQL, the other proceeds

Remember: A deadlock is a circular wait between two transactions — PostgreSQL detects and breaks it by aborting one, unpredictably. The real defense is always acquiring locks on multiple rows/tables in the same, consistent order across every code path; when a deadlock does happen, retry the ENTIRE transaction from its start, not just the failed statement.

See also: transactions and isolation · row level locking · recognizing the pattern

VACUUM and ANALYZE

coreintermediate

PostgreSQL's MVCC never immediately deletes an old row version on UPDATE/DELETE — VACUUM reclaims that dead-tuple space, updates the visibility map, and prevents transaction-id wraparound. VACUUM FULL rewrites the whole table to actually shrink it on disk, but takes an exclusive lock (blocking everything) — plain VACUUM runs alongside normal traffic and is almost always the right one. autovacuum runs both VACUUM and ANALYZE automatically in the background; ANALYZE separately updates the statistics the query planner relies on to choose good plans.

Think of it as

MVCC (multi-version concurrency control) is what lets PostgreSQL show every transaction a consistent snapshot without blocking readers against writers — the price is that an UPDATE doesn't overwrite a row, it creates a NEW version and leaves the old one behind (since some other transaction might still need to see it). VACUUM is the garbage collector for those leftover versions — without it, dead tuples pile up forever, tables bloat, and eventually PostgreSQL would face transaction-id wraparound (a genuinely catastrophic failure mode if truly ignored for billions of transactions). ANALYZE is a completely separate job: sampling the table to keep the query planner's statistics current, which is what lets EXPLAIN produce accurate row-count estimates in the first place.

sql
VACUUM ANALYZE table_name;   -- routine maintenance, safe to run alongside traffic
ANALYZE table_name;           -- refresh planner statistics only, no space reclamation

What we're doing: Refresh a table's statistics after a large bulk data load, so the query planner's next decisions are based on current data rather than stale pre-load statistics.

after_bulk_load.sqlsql
-- after loading a large batch of new rows via COPY or bulk INSERT
COPY orders FROM '/tmp/new_orders.csv' WITH (FORMAT csv);

ANALYZE orders;   -- statistics now reflect the newly-loaded data
3
ANALYZE here specifically refreshes the planner's row-count/distribution statistics — without it, the planner keeps using pre-load statistics that no longer describe the table's actual current size and data distribution, which can lead it to pick a worse plan for subsequent queries.

Why this works: A large bulk load can change a table's size and data distribution dramatically in one operation — autovacuum will eventually catch up on its own schedule, but running ANALYZE explicitly right after a known-large load avoids a window where every query against the freshly-loaded data is planned against stale, pre-load assumptions.

Running VACUUM FULL as routine maintenance instead of plain VACUUM

Wrong

sql
-- a cron job runs this nightly on every table, "just to be thorough"
VACUUM FULL orders;   -- takes ACCESS EXCLUSIVE — blocks reads AND writes for the whole run

Better

sql
-- routine: plain VACUUM (or just let autovacuum handle it)
VACUUM ANALYZE orders;

What you see: A nightly maintenance job causes a real, scheduled outage — every query against that table blocks for the entire duration of VACUUM FULL, which can be a long time on a large table.

Why: VACUUM FULL takes the strongest lock PostgreSQL has (ACCESS EXCLUSIVE), specifically because it physically rewrites the entire table to compact it — that is a fundamentally different, much heavier operation than routine VACUUM, which runs alongside normal traffic using a much weaker lock. VACUUM FULL should be reserved for a table that has genuinely bloated far beyond what routine VACUUM/autovacuum can keep in check, not run as everyday maintenance.

VACUUM vs VACUUM FULL

VACUUM

  • +Non-blocking — normal traffic continues
  • +Reclaims space for reuse within the table
  • +What autovacuum runs — the routine choice

VACUUM FULL

  • ACCESS EXCLUSIVE — blocks everything
  • Actually shrinks the file, returns space to OS
  • Rare — only for a drastically bloated table
  • VACUUM
    • Non-blocking — normal traffic continues
    • Reclaims space for reuse within the table
    • What autovacuum runs — the routine choice
  • VACUUM FULL
    • ACCESS EXCLUSIVE — blocks everything
    • Actually shrinks the file, returns space to OS
    • Rare — only for a drastically bloated table

VACUUM vs VACUUM FULL

VACUUM vs VACUUM FULL
AspectVACUUMVACUUM FULL
Lock takenSHARE UPDATE EXCLUSIVE — normal traffic continuesACCESS EXCLUSIVE — blocks everything
Reclaims space forreuse within the tableactually shrinks the file, returns space to OS
Used by autovacuum?yesnever
When to useroutine, ongoing maintenance (the default choice)rare — a table that is drastically over-bloated

Together

sql
VACUUM ANALYZE orders;      -- routine: reclaim space + refresh planner statistics, non-blocking
VACUUM FULL orders;         -- rare: fully compact the table, but blocks everything while it runs

Remember: VACUUM reclaims dead-tuple space for reuse (non-blocking, routine); VACUUM FULL actually shrinks the table on disk but blocks everything (rare). autovacuum runs both automatically and should almost never be disabled outright. ANALYZE refreshes the query planner's statistics — run it explicitly after a large bulk load rather than waiting for autovacuum's own schedule.

See also: query plans · transactions and isolation · dangerous schema changes

Advertisement

Operating at scale

Connection pooling, replication lag, and table partitioning — three answers to outgrowing a single instance or table.

Connection limits, replicas, and partitioning

standardadvanced

max_connections caps concurrent connections — each one has real memory/process overhead, so raising it arbitrarily high is the wrong fix; a connection pooler (PgBouncer) that multiplexes many app connections onto fewer real PostgreSQL connections is the standard answer. A replica (streaming replication) is a read-only copy of the primary — asynchronous (default, some lag possible) or synchronous (no data loss, slower commits). Partitioning splits one logical table into physically separate pieces (range/list/hash) so queries can skip irrelevant partitions and bulk operations (like dropping old data) become near-instant.

Think of it as

All three of these answer the same underlying question — "what happens once one database instance and one table aren't enough anymore" — from three different angles. Connection limits are about the SERVER's own capacity: every connection reserves real memory whether or not it's doing anything, so a pooler exists to decouple "how many app processes want to talk to the database" from "how many actual database connections exist." Replication is about READ capacity and durability: a hot standby can serve read queries without touching the primary, at the cost of some staleness (async) or some write latency (sync). Partitioning is about a single TABLE outgrowing comfortable size — splitting its rows across physical partitions lets the planner skip entire partitions a query could never match, and lets bulk maintenance (like purging a month of old data) become a near-instant DROP/DETACH instead of a slow, lock-heavy DELETE.

sql
CREATE TABLE t (...) PARTITION BY RANGE (col);
CREATE TABLE t_p1 PARTITION OF t FOR VALUES FROM (x) TO (y);

What we're doing: Partition an events table by month so old data can be purged nearly instantly and queries scoped to a date range skip irrelevant partitions entirely.

events_schema.sqlsql
CREATE TABLE events (id bigint, occurred_at date, payload jsonb)
    PARTITION BY RANGE (occurred_at);

CREATE TABLE events_2026_01 PARTITION OF events FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

SELECT * FROM events WHERE occurred_at >= '2026-01-15' AND occurred_at < '2026-01-20';
-- partition pruning: only events_2026_01 is scanned, every other month's partition is skipped
1
PARTITION BY RANGE (occurred_at) makes events a purely logical container — no rows are stored in the parent table itself, only in its partitions.
5
The query planner recognizes the date range falls entirely within the January partition and skips every other month's partition without even considering it — the more partitions exist, the bigger this pruning win becomes.

Why this works: A single unpartitioned events table with years of history means every query, even one scoped to a single day, has to contend with a much larger index/table than necessary — partitioning by month means a query for "the last week" only ever touches one or two small partitions, and purging data older than a retention window becomes DROP TABLE (near-instant) instead of a DELETE that has to find and remove millions of individual rows.

Raising max_connections instead of adding a connection pooler, to handle more concurrent app instances

Wrong

text
# postgresql.conf
max_connections = 1000   # "just raise it to handle more app servers"

Better

text
# app servers connect to PgBouncer, which multiplexes onto a small real pool
max_connections = 100      # postgresql.conf stays modest
pool_size = 20              # pgbouncer.ini — real connections to PostgreSQL itself

What you see: The database server's memory usage climbs and overall performance degrades as max_connections is raised further, even though most of those connections are idle most of the time — the opposite of the intended fix.

Why: PostgreSQL pre-allocates shared memory and per-backend resources sized to max_connections, whether or not a given connection is actively doing anything — a connection pooler solves the actual underlying problem (many app processes wanting a connection) by letting them share a much smaller number of real, active PostgreSQL connections, instead of paying the per-connection overhead for every one of them simultaneously.

Three ways of outgrowing a single instance/table

Three ways of outgrowing a single instance/table
ProblemAnswer
Too many concurrent app connectionsa connection pooler (PgBouncer), not a higher max_connections
Read load exceeds one server's capacitystreaming replication — hot standby replicas serve reads
One table has grown too large to manage/query efficientlypartitioning (range/list/hash), enabling partition pruning + fast bulk maintenance

Together

sql
CREATE TABLE events (id bigint, occurred_at date, payload jsonb)
    PARTITION BY RANGE (occurred_at);

CREATE TABLE events_2026_01 PARTITION OF events
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

-- dropping a whole month of old data is now near-instant:
DROP TABLE events_2026_01;   -- vs. a slow DELETE ... WHERE occurred_at < ...

Remember: A connection pooler (PgBouncer), not a higher max_connections, is the standard fix for too many concurrent app connections — each connection has real, unavoidable server-side overhead. Async replication (the default) can lag — route read-your-own-write queries to the primary, not a replica. Partitioning splits one large table into physical pieces for partition pruning and near-instant bulk maintenance (DROP/DETACH vs. DELETE).

See also: vacuum and analyze · indexes · query plans

Advertisement