beginnerintermediateadvanced

System Design · Section 70

Cost-Aware Design

Level: intermediate
Read: 15 min
Concepts: 3

A cloud bill is the sum of seven largely independent surfaces — compute, storage, bandwidth, database IOPS, cache memory, cross-region traffic and observability — and the ones engineers forget to model (IOPS, cache memory, observability) frequently outweigh the one they remember (compute), so a cost estimate that stops at compute is an estimate of a smaller, different bill than the one that actually arrives. Turning a total bill into a unit cost — per request, per GB stored, per active user, or per job — is what makes a design's cost legible against growth: a unit cost should be computed from completed useful work, not raw attempts, and any improvement in it should be checked against whether it is a real trend or a one-time step change before it gets projected forward. The last discipline is knowing when to stop: sharding, multi-region active-active and other maximum-scale techniques cost money and operational complexity continuously, starting the day they ship, whether or not the traffic that justifies them ever arrives — moderate scale sized to realistic, measured traffic, with a documented migration trigger for when a real growth signal appears, satisfies the same requirement at a fraction of the cost.

System Design overview

System Design for Beginners CoursefreeCodeCamp.org

What is true here

Seven surfaces bill independently: compute, storage, bandwidth, database IOPS, cache memory, cross-region traffic, observability — review all seven, not just compute.
Unit cost (per request, per GB stored, per active user, per job) is what predicts next month's bill under a traffic change — a total dollar figure cannot.
Compute cost-per-job from completed work, not attempts — a retry-heavy pipeline can hide a rising failure rate behind a flat or falling reported unit cost.
Maximum-scale architecture is a continuous cost paid whether or not the traffic that justifies it ever arrives — size for realistic, measured growth instead.
A documented migration trigger (a latency SLA breach, a DAU threshold) lets a right-sized design scale later without having paid for that capacity the whole time.

What you will be able to do

List the seven resources that independently bill money in a typical cloud architecture, and name which ones are commonly missed
Compute cost per request, per GB stored, per active user and per job from a raw spend figure, and use each to answer a different question
Recognize when a unit-cost improvement is a one-time step change rather than a continuing trend before projecting it forward
Justify an architecture decision against measured, realistic traffic rather than an unquantified maximum-scale scenario

Knowing what costs money

The seven independent surfaces that make up a real cloud bill, and how to turn raw spend into a unit cost that predicts growth.

The cost surface: everything that bills money

coreintermediate

Every design decision has a line item attached to it, and most of those line items are not the one people think about first. Compute (CPU/RAM-hours) and storage (bytes held over time) are the obvious ones — but bandwidth (bytes leaving a cloud, especially across regions or to the public internet), database IOPS (each read/write operation against provisioned-throughput storage, billed separately from the storage itself), cache memory (an in-memory cache like Redis is billed by RAM provisioned, not by data touched), cross-region traffic (replicating data or calling services across regions multiplies both latency and a per-GB transfer charge) and observability (logs, metrics and traces ingested, indexed and retained, often billed per GB per day) all bill independently, and each one can dominate a bill in a design that looks cheap on compute alone. A system can be "efficient" in the metric an engineer optimized for and still be expensive, because a bill is the sum of every resource, not just the one that was watched.

Think of it as

Think of a cloud bill like a restaurant check that itemizes the meal, the drinks, the corkage fee, the delivery fee and the service charge separately — someone who only looks at the menu price of the meal (compute) is repeatedly surprised by a check that is twice as large, because the drinks (storage), corkage (IOPS), delivery (bandwidth) and service charge (observability) were never on their radar as things that cost anything at all. Cross-region traffic is the equivalent of ordering delivery from a restaurant in another city: the food itself may be identical, but now there is a distance-based fee stacked on top that a same-city order never incurs.

text

// A one-page cost-surface checklist to run against any design
// before calling it "done":
Compute:             right-sized instances? autoscaled down off-peak?
Storage:              tiered by access frequency (hot/warm/cold)?
Bandwidth:            served via CDN/cache where possible, not origin?
Database IOPS:        access pattern matches provisioned throughput?
Cache memory:         sized to steady-state, not permanent peak?
Cross-region traffic: is more than one region actually required?
Observability:        log level and retention matched to real need?

What we're doing: Read one month's bill for a service and attribute each line item back to a design decision.

monthly-bill-breakdown.txttext

Service: order-history API, single region, one read replica

Compute (4 always-on instances, sized for peak):   $612
Storage (3 years of orders, no tiering):           $340
Bandwidth (full JSON responses, no CDN):            $205
Database IOPS (provisioned, hot "last 7 days"):     $488
Cache memory (Redis, sized for Black Friday):       $260
Cross-region replica (DR, rarely tested):           $190
Observability (debug logging left on):              $175
------------------------------------------------------
Total:                                             $2,270

3: Instances sized for peak and never scaled down off-peak — a compute cost that looks fixed but is really a scaling decision nobody revisited.
6: IOPS is the single largest line item here, larger even than compute, because the "last 7 days" query pattern is hot against a table holding three years of data with no separate hot/cold split.
7: The cache is sized for a once-a-year peak (Black Friday) but billed at that size continuously — 11 months of the year it is paying for headroom nothing is using.

Why this works: Nothing on this bill is a single "cost" line — it is seven independent charges, and the two biggest ones (IOPS and cache memory) are not the ones most engineers would have guessed before reading it. A design review that only asks "how much compute do we need" misses over half of what this service actually costs.

Treating the compute estimate as the whole cost estimate

Wrong

text

# "Cost estimate" for a new service, written
# before launch, covering only what was easy
# to estimate:
estimated_cost = instances * hourly_rate * hours_per_month
# storage, bandwidth, IOPS, cache, cross-region
# and observability are not mentioned anywhere

Better

text

# Cost estimate itemized across every surface
# that will actually appear on the bill:
estimated_cost = (
    compute_cost +
    storage_cost +          # with tiering assumed
    bandwidth_cost +        # egress, CDN offset
    db_iops_cost +          # hot-path query volume
    cache_memory_cost +     # steady-state sizing
    cross_region_cost +     # only if truly needed
    observability_cost      # log level, retention
)

What you see: The pre-launch cost estimate and the first real invoice disagree by 2-3x, and the gap is never in compute — it shows up as a database-IOPS or observability line item nobody modeled, discovered only after finance asks why the bill jumped.

Why: Compute is the resource every engineer already thinks of as "the cost," because it maps directly onto the mental model of "how many servers do I need" — the other six surfaces bill independently and silently, so an estimate that stops at compute is not an incomplete estimate of the same thing, it is an estimate of a different, smaller bill than the one that will actually arrive.

Seven resources, one bill

Compute

CPU/RAM-hours

Storage

bytes held

Bandwidth

egress bytes

DB IOPS

per operation

Cache memory

RAM provisioned

Cross-region

per-GB transfer

The seven cost surfaces and the failure mode each one causes when ignored

The seven cost surfaces and the failure mode each one causes when ignored
Resource	What is actually billed	Easy-to-miss trap
Compute	CPU/RAM-hours a server or container consumes	Over-provisioned instances running idle most of the day
Storage	Bytes held, often tiered by access frequency	Old data never moved to a cheaper cold tier
Bandwidth	Bytes egressing the cloud or crossing a region boundary	Serving large assets directly from origin instead of a CDN
Database IOPS	Each read/write operation against provisioned throughput	A hot key or chatty query pattern billed per-operation, not per-byte
Cache memory	RAM provisioned for the cache node, continuously	A cache sized for peak traffic left running at that size year-round
Cross-region traffic	Per-GB transfer fee for replication or cross-region calls	A multi-region design chosen before multi-region was actually required
Observability	GB ingested plus GB-day retained for logs/metrics/traces	Debug-level logging or full request tracing left on in production

Remember: A cloud bill is the sum of seven largely independent surfaces — compute, storage, bandwidth, database IOPS, cache memory, cross-region traffic and observability — and the ones people forget (IOPS, cache memory, observability) are frequently larger than the one they remember (compute). Review all seven before calling a cost estimate complete.

Unit-cost estimation: per request, per GB, per user, per job

coreintermediate

A total monthly bill tells you what happened; a unit cost tells you whether it is sustainable as the system grows. Unit-cost estimation takes the total spend on a resource and divides it by the volume of work that resource did, producing a small number attached to something concrete: cost per request (compute plus bandwidth divided by request count), cost per GB stored (storage plus its replication overhead divided by data volume), cost per active user (total spend divided by monthly active users, useful for justifying spend to the business), or cost per job (for batch/async work, total compute divided by jobs completed). The unit number is what actually predicts next month's bill under a traffic change — a total bill by itself does not say whether growth is linear, sub-linear, or dangerously super-linear.

Think of it as

A total bill is like knowing a factory spent $50,000 on materials last month; a unit cost is knowing it costs $2.30 in materials per unit produced. The $50,000 number cannot answer "what happens if we double production" — the $2.30 number answers it directly, and it is also the number that reveals whether producing more units is actually getting cheaper per unit (economies of scale) or more expensive (a bottleneck resource kicking in). Unit cost turns "the bill went up" from a fact into a diagnosis.

text

cost_per_request = (compute_cost + bandwidth_cost + db_cost) / request_count
cost_per_gb      = (storage_cost * replication_factor) / gb_stored
cost_per_user    = total_infra_spend / monthly_active_users
cost_per_job     = total_batch_compute_cost / jobs_completed  # not attempted

What we're doing: Turn one month of raw spend into unit costs, then use them to predict next quarter's bill under 3x growth.

unit-cost-worked-example.txttext

This month:
  Total compute + bandwidth + DB cost:  $18,400
  Total requests served:                15,300,000
  Total GB stored (incl. 3x replication): 2,040 GB
  Storage spend (already includes replication): $184
  Monthly active users:                  30,000
  Batch jobs completed:                  460,000
  Batch compute spend:                   $18,400 (same infra, separate line)

Unit costs:
  cost_per_request = 18400 / 15300000     = $0.0012
  cost_per_gb       = 184 / 2040           = $0.090
  cost_per_user     = 18400 / 30000        = $0.61
  cost_per_job      = 18400 / 460000       = $0.04

Projected at 3x traffic (45.9M requests):
  naive projection (flat unit cost):       $55,200
  if DB IOPS scales worse than linearly:    $70,000+ (measure, don't assume)

6: Storage spend already has the replication factor baked in — dividing by raw (non-replicated) GB stored would understate the true per-GB cost by roughly 3x.
15: The naive projection assumes unit cost stays flat under 3x growth — that assumption is only safe for resources that actually scale linearly; it is exactly the assumption to test against DB IOPS or cache-eviction-rate before trusting it.
17: If a resource scales worse than linearly (a common failure for DB IOPS once a hot table's working set stops fitting in cache), the naive projection understates the real cost — the unit-cost number is only useful once you also know which resources are linear and which are not.

Why this works: This is the actual purpose of unit-cost estimation: not to describe last month, but to project next quarter under a stated growth assumption — something a total dollar figure cannot do on its own, because it carries no information about how spend is distributed across a variable that is about to change (request volume).

Computing cost per job from attempted jobs instead of completed ones

Wrong

text

# A retry-heavy pipeline where 30% of job
# attempts fail and are retried up to 3 times
cost_per_job = total_batch_compute_cost / jobs_attempted
# jobs_attempted includes every retry, so this
# number looks artificially cheap per unit of
# "attempt" while hiding that a third of spend
# produces no completed output at all

Better

text

cost_per_job = total_batch_compute_cost / jobs_completed
# retries are now visible as a cost multiplier
# on the metric that actually matters: work
# that produced a result
retry_overhead_pct = (jobs_attempted - jobs_completed) / jobs_completed

What you see: The reported cost-per-job metric looks stable or even improves over time even as the pipeline's underlying reliability degrades and retry volume climbs — the metric is actively hiding the exact problem (a rising failure rate) that a cost review should have caught first.

Why: Dividing by attempts instead of completions rewards a noisier, less reliable pipeline with an artificially lower unit cost, because failed attempts inflate the denominator without producing anything the business actually wanted — the fix is to always divide by completed, useful output, and track the attempted-vs-completed gap as its own explicit signal.

Same bill, four different unit lenses

Total spend this month

$18,400 total

Divided four ways

$0.0012 / request

$0.09 / GB stored

$0.61 / active user

$0.04 / job

The four standard unit-cost metrics and when each one is the right lens

The four standard unit-cost metrics and when each one is the right lens
Metric	Formula	Best used to answer
Cost per request	(compute + bandwidth + DB/cache) / request count	"What does a traffic spike cost us?"
Cost per GB stored	(storage + replication + backup) / GB stored	"What does retaining more data cost long-term?"
Cost per active user	total infra spend / monthly active users	"Is our margin per user improving as we grow?"
Cost per job	total batch compute / jobs completed (not attempted)	"Is this pipeline getting more or less efficient?"

Remember: A total bill says what happened; a unit cost (per request, per GB stored, per active user, per job) says whether it scales — always divide by completed useful work, include overhead like replication in the numerator, and check whether a unit-cost change is a real trend or a one-time step change before projecting it forward.

Sizing for reality, not hypotheticals

Why maximum-scale architecture is a continuous cost, and how to size for the traffic a product actually has.

Right-sizing, not maximum scale

coreintermediate

Every scaling technique — sharding, multi-region active-active, event-driven microservices, a distributed cache in front of a database that fits in RAM anyway — has a cost that is paid continuously, in dollars and in operational complexity, whether or not the traffic that justifies it ever arrives. Designing for maximum hypothetical scale by default means paying that ongoing cost for capacity the product may never use, while a design sized for the traffic the product actually has (with a clear, cheap path to scale further later) satisfies the same requirements at a fraction of the cost and complexity. The skill is not "always build for scale" or "always build the cheapest thing" — it is estimating the traffic that will realistically arrive in a defined horizon and sizing for that, with headroom, rather than for an unbounded hypothetical.

Think of it as

This is the difference between renting a warehouse sized for the busiest day you can imagine versus one sized for your actual busiest day plus real growth, with a contract that lets you rent more space on short notice. The warehouse sized for an imagined worst case costs full rent every single day of the year, even on the 350 ordinary days when nine-tenths of it sits empty — and the business still has to staff, secure and maintain all of that space regardless of whether it is used. The right-sized warehouse with an expansion option costs less every month and only pays for more space in the months it is actually needed.

text

// A right-sizing checklist before adopting a
// scale-out technique:
1. What is the realistic traffic in 12-18 months,
   not the best-case hypothetical?
2. Does the simpler, cheaper design fail at that
   realistic number, or only at 10-100x it?
3. What is the actual migration cost later, if
   growth exceeds the estimate -- is there a
   clean path, or is it a rewrite?
4. What does the complex design cost every month
   between now and the point growth might justify it?

What we're doing: Compare the two designs from the table on a concrete product: an internal analytics dashboard with 4,000 daily active users.

right-sizing-worked-example.txttext

Product: internal analytics dashboard
Realistic traffic: 4,000 DAU, ~40 req/s peak,
120 GB of data, growing ~15%/year

Option A - built for maximum scale:
  12-shard database cluster, 3-region active-active
  Monthly infra cost:            ~$14,000
  Team needed to operate it:     2 dedicated SREs
  Headroom over realistic peak:  ~500x

Option B - right-sized:
  Single Postgres primary + 1 read replica, one
  region, connection pooling, a documented
  point-in-time-recovery runbook
  Monthly infra cost:            ~$600
  Team needed to operate it:     shared with other services
  Headroom over realistic peak:  ~15x
  Documented migration trigger:  sustained p99 latency
                                 above SLA, or DAU
                                 crossing 50,000

9: Roughly 500x headroom over the realistic peak is not a safety margin, it is capacity that will sit unused for years, if it is ever used at all — every month it is unused, the $14,000 is still being spent.
17: Right-sizing does not mean "no plan for growth" — it means the plan is a documented trigger and a budgeted migration, not pre-paid capacity sitting idle from day one.
18: 15x headroom over a realistic peak, revisited when a concrete signal (latency or DAU) crosses a threshold, satisfies the actual requirement at roughly 4% of the cost of option A.

Why this works: The two options satisfy the identical stated requirement — serving this dashboard's traffic reliably — at a cost difference of more than 20x, because option A sized itself against a traffic level with no supporting evidence rather than the traffic the product actually has. The "moderate scale and lower cost satisfy requirements" principle is exactly this: option B is not a worse system, it is the same system minus capacity nothing is using.

Justifying maximum-scale architecture with "what if we go viral" instead of a measured growth estimate

Wrong

text

# Architecture review justification:
# "We should shard from day one and go
# multi-region, in case we go viral and
# traffic 100x's overnight."
# No current traffic number is cited. No
# growth curve is cited. No cost of the
# alternative (documented migration path)
# is discussed.

Better

text

# Architecture review justification:
# "Current: 4,000 DAU, 40 req/s peak, growing
# 15%/year. Single-region Postgres handles this
# with 15x headroom. Migration to sharding is a
# ~3-week project, triggered by p99 latency
# breaching SLA or DAU crossing 50,000 --
# tracked on a dashboard, reviewed quarterly."

What you see: The team pays a large, continuous infrastructure and operational bill for years for a traffic level that never arrives, while every unrelated feature ships slower because engineers first have to reason about sharding logic, cross-region consistency, or multi-region failover that a simpler design would never have required them to touch.

Why: "What if we go viral" is not a growth estimate — it has no number, no timeframe and no probability attached to it, which makes it unfalsifiable and therefore always available as a justification for any amount of complexity. A measured growth estimate with a stated migration trigger gives the team something to actually check against reality later, and shifts the cost of scale-out to the point where a real signal justifies paying for it, instead of paying for it continuously against a scenario that was never quantified in the first place.

Two designs satisfying the same requirement

Built for maximum hypothetical scale

+Sharded database, multi-region active-active from launch
+Pays full infra cost every month regardless of traffic
+Cross-shard queries and multi-region consistency bugs on-call has to own
+No migration needed if growth ever arrives — but it may never arrive

Right-sized for realistic moderate scale

−Single primary + read replica, single region
−Infra cost matched to actual measured load
−One system to reason about; fewer failure modes
−A budgeted, planned migration path if a real growth signal appears

Same requirement, two architectures, very different ongoing cost

Same requirement, two architectures, very different ongoing cost
Dimension	Built for maximum hypothetical scale	Right-sized for realistic moderate scale
Database	Sharded across 12 nodes from day one	Single primary + read replica, vertically scaled
Deployment	Active-active across 3 regions	Single region, backups + a documented DR runbook
Monthly infra cost	High, paid continuously regardless of traffic	A fraction of the above, matched to actual load
Operational load	Cross-shard queries, multi-region consistency, more on-call surface	One system to reason about, fewer failure modes
Path if traffic exceeds estimate	Already built — no migration needed	A planned, budgeted migration triggered by a real signal

Remember: Scale-out techniques (sharding, multi-region active-active, event-driven fan-out) cost money and operational complexity every month starting the day they ship — pay that cost when a measured growth signal justifies it, not against an unquantified "what if we go viral" scenario. Moderate scale sized to realistic traffic, with a documented migration trigger, satisfies the same requirement at a fraction of the cost.