Filter concepts by levelShowing all levels.

MongoDB · Section 16

Aggregation Framework

Level
advanced
Read
35 min
Concepts
6

MongoDB's primary analytical and transformation tool: a pipeline of named stages ($match, $project/$set/$unset, $group, $sort/$limit/$skip, $facet, and the specialized $count/$sample/$bucket family), the ordering principles that determine real cost independent of logical correctness, pushing a selective $match as early as possible so it can use an index exactly like find(), the 100MB-per-stage memory limit and automatic disk-spilling behavior, and recognizing when a summary-shaped computation belongs in the pipeline rather than an application-side fetch-and-loop.

What is true here

  1. aggregate([stage1, stage2, ...]) runs stages in sequence, each transforming the previous stage's output — find() answers "which," aggregate() answers "transformed how."
  2. The core stage vocabulary: $match, $project/$set/$unset, $group, $sort/$limit/$skip, $facet, $count/$sample/$bucket family — $unwind, $lookup, $setWindowFields get their own deeper sections.
  3. Two logically identical pipelines can have very different real cost purely from stage order — filter and reshape as early as possible.
  4. A leading $match can use an index exactly like find(); one placed after a reshaping stage filters transformed data and cannot.
  5. Stages that must hold state in memory are capped at 100MB by default, auto-spilling to disk since MongoDB 6.0 rather than erroring outright.

What you will be able to do

  • Build a multi-stage aggregation pipeline from the core stage vocabulary
  • Reorder a pipeline's stages to reduce real cost while preserving its logical result
  • Place $match early enough that it can use an index, and confirm it with explain()
  • Predict when a pipeline stage risks the 100MB memory limit and understand the disk-spilling consequence
  • Recognize a summary-shaped computation and choose aggregation over an application-side fetch-and-loop

The pipeline and its vocabulary

What a pipeline is, and the core stage vocabulary every aggregation is built from.

Aggregation as a pipeline of stages

coreintermediate

db.collection.aggregate([...]) runs an array of stages in sequence — each stage takes the previous stage's output as its input, transforms or filters it, and passes results to the next. It is MongoDB's tool for grouping, reshaping, and computing across documents, beyond what find() alone can do.

Think of it as

Think of an aggregation pipeline the way you would a Unix pipe chain (grep | sort | uniq) — each stage does one job, and the whole pipeline's behavior comes from composing simple stages in sequence, not from one stage doing everything. find() answers "which documents match" — aggregate() answers "what does the data look like once transformed, grouped, or computed."

text
db.<collection>.aggregate([ { $stage1: {...} }, { $stage2: {...} }, ... ])

What we're doing: Show a short real pipeline and trace what each stage contributes to the final shape.

basic-pipeline.txttext
db.orders.aggregate([
  { $match: { status: "delivered" } },
  { $group: { _id: "$customerId", total: { $sum: "$amount" } } },
  { $sort: { total: -1 } }
])
2
$match filters down to delivered orders only — the first stage narrows the working set before anything more expensive runs.
4
$group reshapes the documents entirely — the output is now one document per customer, not one per order, with a computed total.

Why this works: The pipeline shape (filter, then group, then sort) mirrors how a human would describe the task in words — "delivered orders, totaled per customer, biggest spenders first" — which is what makes aggregation pipelines readable despite doing real analytical work.

Trying to do multi-document aggregation work in application code instead of the pipeline

Wrong

text
// Fetching every delivered order with find(), then looping in application code to sum totals per customer

Better

text
// db.orders.aggregate([{ $match: {...} }, { $group: {...} }]) — let the database do the grouping and summing

What you see: The application transfers far more data over the network than it needs (every raw order document) and spends CPU time doing work the database is built to do more efficiently.

Why: find() only filters and shapes individual documents — grouping, summing, and other cross-document computation is exactly what the aggregation pipeline exists for, and doing it in the database avoids transferring raw data just to immediately reduce it in application code.

Documents flow through stages, in order

collection

input documents

stage 1, stage 2, ...

each transforms the last

result

transformed output

  1. collection — input documents
  2. stage 1, stage 2, ... — each transforms the last
  3. result — transformed output

Remember: aggregate([stage1, stage2, ...]) runs stages in sequence, each transforming the previous stage's output. find() answers "which documents match"; aggregate() answers "what does the data look like transformed, grouped, or computed."

See also: core pipeline stages · pipeline ordering principles

The core pipeline stages

coreintermediate

A vocabulary of named stages, each with one job: $match filters, $project/$set/$unset reshape fields, $group aggregates, $sort/$limit/$skip order and page, $facet runs several sub-pipelines at once, $count/$sample/$bucket/$bucketAuto/$sortByCount are specialized summarizers. $unwind, $lookup, and $setWindowFields get their own full sections (18, 19, 20).

Think of it as

Each stage name describes what it does almost literally — $match matches, $sort sorts, $count counts — which is why a pipeline built from several of them reads close to a sentence describing the task. Learning the vocabulary is mostly learning to recognize which single-purpose stage a given transformation need maps to.

text
{ $match: {...} } · { $project: {...} } · { $set: {...} } · { $group: {...} } · { $sort: {...} } · { $limit: N }

What we're doing: Use $replaceRoot and $facet to show two stages that reshape more dramatically than $project alone.

replaceroot-and-facet.txttext
// $replaceRoot: promote a nested sub-document to be the whole output document
db.orders.aggregate([{ $replaceRoot: { newRoot: "$shippingAddress" } }])

// $facet: two independent summaries of the same input, in one round trip
db.orders.aggregate([{ $facet: {
  byStatus: [{ $group: { _id: "$status", count: { $sum: 1 } } }],
  byRegion: [{ $group: { _id: "$region", count: { $sum: 1 } } }]
} }])
2
$replaceRoot discards everything outside shippingAddress, making it the entire new document — a more drastic reshape than $project's field-by-field control.
5
$facet runs both sub-pipelines against the exact same input documents and returns both results together, avoiding two separate aggregate() calls.

Why this works: These two stages solve problems the simpler stages ($match, $project, $group alone) cannot — $replaceRoot for genuinely restructuring the document shape, $facet for needing multiple independent summaries without re-querying the collection multiple times.

Running several separate aggregate() calls for related summaries instead of one $facet pipeline

Wrong

text
// Two separate aggregate() round trips: one grouped by status, one grouped by region, both scanning the same collection

Better

text
// One aggregate() call using $facet, both summaries computed from a single pass over the input

What you see: A dashboard or report makes several near-identical aggregate() calls against the same collection and filter, each re-scanning the same data independently.

Why: $facet exists specifically to answer "give me several different summaries of the same filtered input" in one pipeline execution, rather than paying the cost of scanning and filtering the same documents multiple times across separate calls.

A pipeline built entirely from named stages

$match

filter

$group

aggregate by key

$sort

order results

$limit

page the output

  • $match — filter
    • leads to $group
  • $group — aggregate by key
    • leads to $sort
  • $sort — order results
    • leads to $limit
  • $limit — page the output

The core stage vocabulary

The core stage vocabulary
StageJob
$matchfilter documents, same syntax as find()
$projectreshape output fields explicitly
$set / $addFieldsadd or compute fields without redeclaring the rest
$unsetremove fields
$groupcollapse documents into groups, with accumulators
$sort / $limit / $skiporder and page results
$replaceRoot / $replaceWithreplace the document with a specified sub-document
$facetrun several sub-pipelines on the same input, side by side
$countoutput a single document with the count of input documents
$samplerandomly select N documents
$bucket / $bucketAutogroup documents into ranges (manual or automatic boundaries)
$sortByCountgroup by a field and sort by group size, in one stage

Together

text
db.orders.aggregate([
  { $match: { status: "delivered" } },
  { $group: { _id: "$region", revenue: { $sum: "$amount" }, count: { $sum: 1 } } },
  { $sort: { revenue: -1 } },
  { $limit: 5 }
])
// -> top 5 regions by delivered revenue, in one pipeline built entirely from this table's stages

Remember: $match filters, $project/$set/$unset reshape fields, $group aggregates, $sort/$limit/$skip order and page, $facet runs parallel sub-pipelines, $count/$sample/$bucket/$bucketAuto/$sortByCount specialize further. $unwind, $lookup, $setWindowFields get their own sections.

See also: aggregation as a pipeline · pipeline ordering principles

Advertisement

Ordering, cost, and when to reach for it

The ordering principles that determine real cost, pushing $match early specifically, the memory limit, and choosing aggregation over an application-side loop.

Pipeline ordering principles

standardintermediate

Stage order changes both correctness and cost: reduce the document count as early as possible (filter before transforming), and reshape only what later stages actually need. The same logical result can come from pipelines with very different real costs depending on stage order.

Think of it as

Think of each stage as doing work proportional to how many documents pass through it — a stage placed early, before the document count shrinks, pays that cost on every document; the same stage placed after a filtering stage pays it only on the survivors. Ordering a pipeline well means asking "does this stage need to see all the documents, or only the ones a later stage would keep anyway."

text
// General order: $match (filterable, indexed) -> $project (drop unneeded fields) -> $group/$sort -> $limit

What we're doing: Contrast two logically equivalent pipelines with very different real cost, based purely on stage order.

reordering-for-cost.txttext
// Expensive: groups ALL orders first, then filters groups down to one region
db.orders.aggregate([
  { $group: { _id: "$region", total: { $sum: "$amount" } } },
  { $match: { _id: "west" } }
])

// Cheap: filters to one region's orders first, then groups only those
db.orders.aggregate([
  { $match: { region: "west" } },
  { $group: { _id: "$region", total: { $sum: "$amount" } } }
])
2
This pipeline computes a $sum for every region, across the entire collection, only to discard every result except one — most of that grouping work is wasted.
7
Filtering to the one relevant region first means $group only ever processes the documents that could contribute to the final answer.

Why this works: Both pipelines produce the identical final result — this is purely a cost difference, and it is the clearest possible illustration of why "logically correct" and "well-ordered" are different standards for a pipeline to meet.

Writing a pipeline in the order the requirements were described, rather than the order that minimizes work

Wrong

text
// "Group orders by region, then only show west" -> written as $group, then $match, matching the sentence order

Better

text
// Reorder to $match first, $group second — same logical result, but only the relevant documents get grouped

What you see: A pipeline that "makes sense" in the order it was described performs far worse than an equivalent pipeline with stages reordered for cost.

Why: A pipeline's stage order does not have to match the order a requirement was verbally described in — the only constraint is that later stages still see the data they need, and within that constraint, moving filtering earlier is close to always a free performance win.

Remember: Filter as early as possible; project away unneeded fields before expensive stages; put $sort before $limit for the top-N optimization. Two pipelines can be logically identical but have very different real cost based purely on stage order.

See also: core pipeline stages · push match early

Pushing $match as early as practical

coreintermediate

A $match placed first in a pipeline, on an indexed field, can use that index the way find() would — turning the whole pipeline's starting point into an IXSCAN instead of a full collection scan. A $match placed after other stages cannot use an index, because it is filtering already-transformed intermediate documents, not the original collection.

Think of it as

An index belongs to the collection's real documents, not to whatever an aggregation pipeline has transformed them into partway through — so a $match can only use an index if it is looking at the original, untransformed documents, which means it has to be at or very near the start of the pipeline. This is the direct aggregation-pipeline analog of everything sections 10-15 already established about find() and indexes.

text
db.<collection>.aggregate([{ $match: {...} }, ...])   // as close to first as the logic allows

What we're doing: Confirm with explain() that a leading $match uses an index, exactly the way find() does.

match-uses-index-in-pipeline.txttext
db.orders.createIndex({ status: 1 })

db.orders.aggregate([{ $match: { status: "pending" } }, { $group: {...} }]).explain()
// -> the $match stage's own explain shows "IXSCAN" on the status index — same mechanism as find({ status: "pending" })
1
This is the exact same index find() would use — an aggregation pipeline's leading $match is not a different mechanism, just a different call shape.
4
explain() on the aggregate() call shows the same stage-tree vocabulary (IXSCAN) as find()'s explain() — confirming the index really is used at this specific point in the pipeline.

Why this works: This equivalence — a leading $match using an index exactly like find() — is what makes "push $match early" a concrete, checkable optimization rather than a vague best practice: it is literally the difference between an indexed lookup and a full scan, verifiable the same way as any find() query.

Placing $match after a $project or $group stage that already discarded or transformed the field being matched on

Wrong

text
db.orders.aggregate([{ $project: { region: 1, amount: 1 } }, { $match: { status: "pending" } }])  // status was dropped by $project

Better

text
db.orders.aggregate([{ $match: { status: "pending" } }, { $project: { region: 1, amount: 1 } }])  // match first, then reshape

What you see: The pipeline fails to filter as expected, or worse, needs to keep the field around through $project just so a later $match can still find it — both losing the index benefit and adding avoidable complexity.

Why: Once a stage reshapes or drops a field, a later $match on that field is working with different data than the collection's indexed field — at best it loses the index, at worst the field no longer exists in the expected shape at all.

$match at the start can use an index; later, it cannot

$match first

sees real documents, index-eligible

$group / $project

reshapes into something new

$match after

filters reshaped output, no index

  1. $match first — sees real documents, index-eligible
  2. $group / $project — reshapes into something new
  3. $match after — filters reshaped output, no index

Remember: A $match at (or near) the start of a pipeline can use an index exactly like find() — verified via explain(). A $match after a reshaping stage filters transformed intermediate documents and cannot use the collection's index.

See also: pipeline ordering principles · collscan vs ixscan

Pipeline memory and intermediate-result limits

coreadvanced

Each stage that must hold data in memory (like $group or $sort without an index) is capped at 100MB by default. Exceeding it either fails the query or spills to disk, depending on allowDiskUse — automatically enabled by default since MongoDB 6.0, but disk spilling is slower than staying in memory.

Think of it as

Some stages can stream — process one document, pass it on, forget it — but others (like $group, which must see every document in a group before it knows the final total) have to hold their working state in memory until they have processed everything. The 100MB limit exists because that held-in-memory state is a real resource cost, per stage, not a soft guideline.

text
db.<collection>.aggregate([...], { allowDiskUse: true })   // explicit override, though 6.0+ defaults to true already

What we're doing: Show a $group stage over a large collection needing disk spilling, and how to recognize that in explain()/execution stats.

memory-limit-and-spilling.txttext
db.events.aggregate([
  { $group: { _id: "$userId", eventCount: { $sum: 1 } } }
])
// -> on a large collection with millions of distinct userId values, the $group stage's
//    working set can exceed 100MB and spill to disk automatically (MongoDB 6.0+)
1
A $group with high-cardinality _id (many distinct userId values) has to hold a running total for every distinct value simultaneously — this is exactly the shape of workload that can exceed 100MB.
5
Automatic disk spilling means this query succeeds rather than erroring outright, but it runs slower than an equivalent group with a smaller working set — worth noticing, not just tolerating.

Why this works: Understanding which specific stages carry this risk (the ones that must hold state, not the ones that stream) is what lets someone predict, before running a pipeline, whether a given $group or $sort is a candidate for hitting the memory limit on a large enough collection.

Relying on automatic disk spilling as a substitute for a well-designed pipeline

Wrong

text
// A $group over a very high-cardinality field on a huge collection, tolerated because it "still works" via disk spilling

Better

text
// Filter with an early $match to reduce cardinality first, or reconsider whether the full group is actually needed for this query

What you see: A pipeline that "works" (does not error) is nonetheless noticeably slower than expected, and the cause is disk I/O from repeated spilling rather than an outright failure.

Why: Disk spilling exists to prevent a hard failure, not to make an oversized in-memory operation free — a stage that spills is still doing real, slower disk-bound work, which is exactly the kind of cost pipeline ordering (pushing $match early to reduce cardinality first) is meant to avoid.

A stage that must hold state, past its limit
yes, diskuse allowedyes, diskuse disabled

$group / $sort

holds working state in memory

> 100MB?

per-stage limit

Spill to disk

allowDiskUseByDefault (6.0+)

Error

if disk use is disabled

  • $group / $sort — holds working state in memory
    • leads to > 100MB?
  • > 100MB? — per-stage limit
    • leads to Spill to disk (yes, disk use allowed)
    • leads to Error (yes, disk use disabled)
  • Spill to disk — allowDiskUseByDefault (6.0+)
  • Error — if disk use is disabled

Remember: A stage that must hold state in memory (mainly $group, $sort without an index, $bucket family) is capped at 100MB by default, per stage. MongoDB 6.0+ auto-spills to disk rather than erroring, but spilling is slower — a signal to optimize, not just tolerate.

See also: push match early · pipeline ordering principles

Aggregation vs. application-side loops

standardintermediate

Whenever a computation only needs a summary of the data — a total, a count, a top-N, a grouping — do it in the pipeline rather than fetching every raw document and looping in application code. The pipeline avoids transferring data the loop would immediately discard, and the database is built to do this work efficiently.

Think of it as

An application-side loop over fetched documents is doing, by hand, exactly what a pipeline stage already does natively — the difference is that the loop first has to pay the cost of transferring every raw document over the network, one document at a time, before any of that reduction work can even begin. The pipeline does the reduction where the data already lives.

text
// If the end result is smaller than the input (a count, a sum, a top-N) — that reduction belongs in the pipeline, not a loop

What we're doing: Contrast fetching-and-looping against an equivalent aggregation for the same summary, quantifying the data-transfer difference.

loop-vs-aggregate.txttext
// Application loop: fetch every delivered order, sum in code
const orders = await db.collection("orders").find({ status: "delivered" }).toArray()
const total = orders.reduce((sum, o) => sum + o.amount, 0)
// -> transfers every full order document just to compute one number

// Aggregation: the same result, computed where the data lives
const [{ total }] = await db.collection("orders").aggregate([
  { $match: { status: "delivered" } },
  { $group: { _id: null, total: { $sum: "$amount" } } }
]).toArray()
1
Every field of every delivered order crosses the network here, even though the application only ever uses the amount field, and only to add it up.
5
The aggregation computes the same single number without transferring any full document — only the final, tiny result crosses the network.

Why this works: This example makes the cost difference concrete: the application loop pays a network-transfer cost proportional to the full collection size, while the aggregation pays a cost proportional only to the final summary size — a gap that grows directly with how much the operation actually reduces the data.

Defaulting to find() + application-side reduction out of habit, even for genuinely summary-shaped results

Wrong

text
// find({...}).toArray() followed by .reduce()/.filter()/.map() in application code, for a task that produces a much smaller summary

Better

text
// Recognize the summary shape and reach for $group/$count/$sum in the pipeline instead — same result, far less data movement

What you see: An endpoint or report becomes measurably slower as the underlying collection grows, tracing back to fetching and reducing full documents in application code for what is fundamentally a summary computation.

Why: The cost of fetch-then-reduce in application code grows with the full input size, while the cost of an equivalent aggregation grows with the final result size — for any genuinely summary-shaped task, that gap widens as the collection grows, making the choice matter more over time, not less.

Remember: If the result is a summary (count, sum, grouped totals, top-N) — aggregate it in the pipeline, not a fetch-and-loop in application code. The pipeline avoids transferring raw data the loop would immediately reduce away.

See also: aggregation as a pipeline · push match early

Advertisement