beginnerintermediateadvanced

MongoDB · Aggregation Framework

Aggregation Framework

Pipelines of stages, ordering for real cost, memory limits, and when to aggregate over looping

Stages, in sequenceeach transforms the previous stage's output
Filter as early as possiblea leading $match can use an index, like find()
100MB per in-memory stageauto-spills to disk since MongoDB 6.0
Summary-shaped? Aggregate.not fetch-and-loop in application code

1.Aggregation as a pipeline of stages

db.collection.aggregate([...]) runs an array of stages in sequence — each stage takes the previous stage's output as its input, transforms or filters it, and passes results to the next. It is MongoDB's tool for grouping, reshaping, and computing across documents, beyond what find() alone can do.

Key features

db.collection.aggregate([ stage1, stage2, ... ]) — an array of stage objects, in order
Each stage's output becomes the next stage's input — documents flow through, not processed all-at-once
A stage can filter documents out, create new ones, or reshape them entirely — not always one-in-one-out
Most stages can repeat in a pipeline; $out, $merge, and $geoNear cannot appear more than once
find() answers "which documents match"; aggregate() answers "what does the data look like transformed/grouped/computed"

Documents flow through stages, in order

collection

input documents

stage 1, stage 2, ...

each transforms the last

result

transformed output

Command

text

db.<collection>.aggregate([ { $stage1: {...} }, { $stage2: {...} }, ... ])

aggregate([stage1, stage2, ...]) runs stages in sequence, each transforming the previous stage's output. find() answers "which documents match"; aggregate() answers "what does the data look like transformed, grouped, or computed."

2.The core pipeline stages

A vocabulary of named stages, each with one job: $match filters, $project/$set/$unset reshape fields, $group aggregates, $sort/$limit/$skip order and page, $facet runs several sub-pipelines at once, $count/$sample/$bucket/$bucketAuto/$sortByCount are specialized summarizers. $unwind, $lookup, and $setWindowFields get their own full sections (18, 19, 20).

Key features

$match: filters documents by a query-like condition, same syntax as find()
$project: reshapes output — include/exclude/compute fields explicitly
$set (alias addFields) / $unset: add or remove fields without listing every other field
$group: collapses documents into groups by a key, computing accumulators ($sum, $avg, ...) per group
$sort / $limit / $skip: order results and page through them, same semantics as find()
$facet, $count, $sample, $bucket, $bucketAuto, $sortByCount: parallel sub-pipelines and specialized summarizers

The core stage vocabulary

The core stage vocabulary
Stage	Job
$match	filter documents, same syntax as find()
$project	reshape output fields explicitly
$set / $addFields	add or compute fields without redeclaring the rest
$unset	remove fields
$group	collapse documents into groups, with accumulators
$sort / $limit / $skip	order and page results
$replaceRoot / $replaceWith	replace the document with a specified sub-document
$facet	run several sub-pipelines on the same input, side by side
$count	output a single document with the count of input documents
$sample	randomly select N documents
$bucket / $bucketAuto	group documents into ranges (manual or automatic boundaries)
$sortByCount	group by a field and sort by group size, in one stage

A pipeline built entirely from named stages

$match

filter

$group

aggregate by key

$sort

order results

$limit

page the output

Command

text

{ $match: {...} } · { $project: {...} } · { $set: {...} } · { $group: {...} } · { $sort: {...} } · { $limit: N }

$match filters, $project/$set/$unset reshape fields, $group aggregates, $sort/$limit/$skip order and page, $facet runs parallel sub-pipelines, $count/$sample/$bucket/$bucketAuto/$sortByCount specialize further. $unwind, $lookup, $setWindowFields get their own sections.

3.Pipeline ordering principles

Stage order changes both correctness and cost: reduce the document count as early as possible (filter before transforming), and reshape only what later stages actually need. The same logical result can come from pipelines with very different real costs depending on stage order.

Key features

Filter (reduce document count) as early as the query logic allows — every later stage processes fewer documents
Project away unneeded fields early if a later, expensive stage ($group, $sort) does not need them
A stage that can use an index (like $match on an indexed field) benefits most from running first, before other stages reshape the documents
$sort before $limit lets MongoDB potentially avoid a full sort (top-N optimization) — the reverse order cannot
Ordering is a performance concern, not usually a correctness one — most stage reorderings that preserve logical meaning still change real cost

Command

text

// General order: $match (filterable, indexed) -> $project (drop unneeded fields) -> $group/$sort -> $limit

Filter as early as possible; project away unneeded fields before expensive stages; put $sort before $limit for the top-N optimization. Two pipelines can be logically identical but have very different real cost based purely on stage order.

4.Pushing $match as early as practical

A $match placed first in a pipeline, on an indexed field, can use that index the way find() would — turning the whole pipeline's starting point into an IXSCAN instead of a full collection scan. A $match placed after other stages cannot use an index, because it is filtering already-transformed intermediate documents, not the original collection.

Key features

A $match at the very start of a pipeline can use an index exactly like an equivalent find() call
A $match placed after $group, $project, or other reshaping stages filters intermediate results, and cannot use a collection index
MongoDB's own optimizer will move some $match stages earlier automatically, but this is an internal optimization, not something to rely on over explicit good ordering
A highly selective early $match (few matches, e.g. a rare status) gives the largest downstream benefit — every later stage processes far fewer documents
explain() on an aggregate() call shows the same stage tree concepts (IXSCAN, COLLSCAN) as it does for find()

$match at the start can use an index; later, it cannot

$match first

sees real documents, index-eligible

$group / $project

reshapes into something new

$match after

filters reshaped output, no index

Command

text

db.<collection>.aggregate([{ $match: {...} }, ...])   // as close to first as the logic allows

A $match at (or near) the start of a pipeline can use an index exactly like find() — verified via explain(). A $match after a reshaping stage filters transformed intermediate documents and cannot use the collection's index.

5.Pipeline memory and intermediate-result limits

Each stage that must hold data in memory (like $group or $sort without an index) is capped at 100MB by default. Exceeding it either fails the query or spills to disk, depending on allowDiskUse — automatically enabled by default since MongoDB 6.0, but disk spilling is slower than staying in memory.

Key features

Default limit: 100MB per stage that must hold data in memory — not a whole-pipeline limit, a per-stage one
$group, $bucket, $bucketAuto, $sort (when not index-supported), $sortByCount, $setWindowFields are the stages that can hit it
MongoDB 6.0+: allowDiskUseByDefault is true — exceeding stages write to disk automatically unless explicitly disabled
Disk spilling avoids the hard error but is slower than staying in memory — a signal to optimize the pipeline, not just a safety net to rely on
$search is exempt from the 100MB limit because it runs in a separate process

A stage that must hold state, past its limit

yes, diskuse allowedyes, diskuse disabled

$group / $sort

holds working state in memory

> 100MB?

per-stage limit

Spill to disk

allowDiskUseByDefault (6.0+)

Error

if disk use is disabled

Command

text

db.<collection>.aggregate([...], { allowDiskUse: true })   // explicit override, though 6.0+ defaults to true already

A stage that must hold state in memory (mainly $group, $sort without an index, $bucket family) is capped at 100MB by default, per stage. MongoDB 6.0+ auto-spills to disk rather than erroring, but spilling is slower — a signal to optimize, not just tolerate.

6.Aggregation vs. application-side loops

Whenever a computation only needs a summary of the data — a total, a count, a top-N, a grouping — do it in the pipeline rather than fetching every raw document and looping in application code. The pipeline avoids transferring data the loop would immediately discard, and the database is built to do this work efficiently.

Key features

If the output is a summary (count, sum, average, grouped totals), aggregation is almost always the right tool
Application-side looping means transferring every raw document, even though most of it is immediately reduced away
Aggregation lets an early $match use an index — a loop has already fetched everything before any filtering logic runs in application code
A loop still makes sense for per-document business logic that genuinely needs to run in application code (e.g. calling an external API per document)
A hybrid approach is common: aggregate to reduce/summarize, then loop over the much smaller result set for anything that needs application-side logic

Command

text

// If the end result is smaller than the input (a count, a sum, a top-N) — that reduction belongs in the pipeline, not a loop

If the result is a summary (count, sum, grouped totals, top-N) — aggregate it in the pipeline, not a fetch-and-loop in application code. The pipeline avoids transferring raw data the loop would immediately reduce away.

Spotted a mistake on this page? Tell us.

Read Sheet