beginnerintermediateadvanced

System Design · Cost-Aware Design

Cost-Aware Design

Every resource that bills money, how to turn a bill into a unit cost, and why maximum scale is not the default goal

Seven cost surfacescompute, storage, bandwidth, IOPS, cache, cross-region, observability
Unit costper request, per GB, per user, per job
Right-sizingmoderate scale at lower cost beats unused headroom

1.The cost surface: everything that bills money

Every design decision has a line item attached to it, and most of those line items are not the one people think about first. Compute (CPU/RAM-hours) and storage (bytes held over time) are the obvious ones — but bandwidth (bytes leaving a cloud, especially across regions or to the public internet), database IOPS (each read/write operation against provisioned-throughput storage, billed separately from the storage itself), cache memory (an in-memory cache like Redis is billed by RAM provisioned, not by data touched), cross-region traffic (replicating data or calling services across regions multiplies both latency and a per-GB transfer charge) and observability (logs, metrics and traces ingested, indexed and retained, often billed per GB per day) all bill independently, and each one can dominate a bill in a design that looks cheap on compute alone. A system can be "efficient" in the metric an engineer optimized for and still be expensive, because a bill is the sum of every resource, not just the one that was watched.

Key features

Compute and storage are the intuitive costs; bandwidth, DB IOPS, cache memory, cross-region traffic and observability are the ones teams routinely forget to model
Bandwidth is typically metered on egress (data leaving the cloud or crossing a region boundary), not ingress — pulling data in is usually free, pushing it out is not
Provisioned-throughput databases (e.g. DynamoDB, some managed Postgres/MySQL tiers) bill IOPS separately from the storage bytes — a small table with a hot read pattern can cost more than a huge table that is rarely touched
An in-memory cache like Redis or Memcached is billed by RAM provisioned for the node, continuously, regardless of hit rate — an oversized or underused cache burns money whether or not it is helping
Cross-region replication or cross-region service calls stack a per-GB transfer fee on top of the compute/storage cost that would exist in a single region
Observability (logs, metrics, traces) is billed per GB ingested and per GB-day retained in most vendors — verbose debug logging left on in production is a real, recurring, and frequently invisible cost

The seven cost surfaces and the failure mode each one causes when ignored

The seven cost surfaces and the failure mode each one causes when ignored
Resource	What is actually billed	Easy-to-miss trap
Compute	CPU/RAM-hours a server or container consumes	Over-provisioned instances running idle most of the day
Storage	Bytes held, often tiered by access frequency	Old data never moved to a cheaper cold tier
Bandwidth	Bytes egressing the cloud or crossing a region boundary	Serving large assets directly from origin instead of a CDN
Database IOPS	Each read/write operation against provisioned throughput	A hot key or chatty query pattern billed per-operation, not per-byte
Cache memory	RAM provisioned for the cache node, continuously	A cache sized for peak traffic left running at that size year-round
Cross-region traffic	Per-GB transfer fee for replication or cross-region calls	A multi-region design chosen before multi-region was actually required
Observability	GB ingested plus GB-day retained for logs/metrics/traces	Debug-level logging or full request tracing left on in production

Seven resources, one bill

Compute

CPU/RAM-hours

Storage

bytes held

Bandwidth

egress bytes

DB IOPS

per operation

Cache memory

RAM provisioned

Cross-region

per-GB transfer

Command

text

// A one-page cost-surface checklist to run against any design
// before calling it "done":
Compute:             right-sized instances? autoscaled down off-peak?
Storage:              tiered by access frequency (hot/warm/cold)?
Bandwidth:            served via CDN/cache where possible, not origin?
Database IOPS:        access pattern matches provisioned throughput?
Cache memory:         sized to steady-state, not permanent peak?
Cross-region traffic: is more than one region actually required?
Observability:        log level and retention matched to real need?

A cloud bill is the sum of seven largely independent surfaces — compute, storage, bandwidth, database IOPS, cache memory, cross-region traffic and observability — and the ones people forget (IOPS, cache memory, observability) are frequently larger than the one they remember (compute). Review all seven before calling a cost estimate complete.

2.Unit-cost estimation: per request, per GB, per user, per job

A total monthly bill tells you what happened; a unit cost tells you whether it is sustainable as the system grows. Unit-cost estimation takes the total spend on a resource and divides it by the volume of work that resource did, producing a small number attached to something concrete: cost per request (compute plus bandwidth divided by request count), cost per GB stored (storage plus its replication overhead divided by data volume), cost per active user (total spend divided by monthly active users, useful for justifying spend to the business), or cost per job (for batch/async work, total compute divided by jobs completed). The unit number is what actually predicts next month's bill under a traffic change — a total bill by itself does not say whether growth is linear, sub-linear, or dangerously super-linear.

Key features

Cost per request = (compute + bandwidth + relevant DB/cache cost over a period) / request count in that period — the number that predicts what a traffic spike costs
Cost per GB stored should include replication and backup overhead, not just raw bytes — three-way replication makes the true per-GB cost roughly 3x the storage sticker price
Cost per active user is a business-facing metric (total infra spend / monthly active users) — useful for comparing against revenue per user, not for predicting the effect of a specific engineering change
Cost per job (batch/async work) should be measured per completed job, not per attempted job — retries and failures inflate compute spend without producing output, and hiding that inflates the apparent unit cost of legitimate work
A flat or falling unit cost as volume grows is a healthy sign (economies of scale, better amortization of fixed costs); a rising unit cost as volume grows means some resource is scaling worse than linearly and needs to be found before it becomes the whole bill

The four standard unit-cost metrics and when each one is the right lens

The four standard unit-cost metrics and when each one is the right lens
Metric	Formula	Best used to answer
Cost per request	(compute + bandwidth + DB/cache) / request count	"What does a traffic spike cost us?"
Cost per GB stored	(storage + replication + backup) / GB stored	"What does retaining more data cost long-term?"
Cost per active user	total infra spend / monthly active users	"Is our margin per user improving as we grow?"
Cost per job	total batch compute / jobs completed (not attempted)	"Is this pipeline getting more or less efficient?"

Same bill, four different unit lenses

Total spend this month

$18,400 total

Divided four ways

$0.0012 / request

$0.09 / GB stored

$0.61 / active user

$0.04 / job

Command

text

cost_per_request = (compute_cost + bandwidth_cost + db_cost) / request_count
cost_per_gb      = (storage_cost * replication_factor) / gb_stored
cost_per_user    = total_infra_spend / monthly_active_users
cost_per_job     = total_batch_compute_cost / jobs_completed  # not attempted

A total bill says what happened; a unit cost (per request, per GB stored, per active user, per job) says whether it scales — always divide by completed useful work, include overhead like replication in the numerator, and check whether a unit-cost change is a real trend or a one-time step change before projecting it forward.

3.Right-sizing, not maximum scale

Every scaling technique — sharding, multi-region active-active, event-driven microservices, a distributed cache in front of a database that fits in RAM anyway — has a cost that is paid continuously, in dollars and in operational complexity, whether or not the traffic that justifies it ever arrives. Designing for maximum hypothetical scale by default means paying that ongoing cost for capacity the product may never use, while a design sized for the traffic the product actually has (with a clear, cheap path to scale further later) satisfies the same requirements at a fraction of the cost and complexity. The skill is not "always build for scale" or "always build the cheapest thing" — it is estimating the traffic that will realistically arrive in a defined horizon and sizing for that, with headroom, rather than for an unbounded hypothetical.

Key features

Maximum-scale architecture (sharding, multi-region active-active, event sourcing at every layer) is a form of complexity and cost paid continuously, starting the day it ships, regardless of whether the traffic that justifies it ever appears
A vertically-scaled single database, or a single-region deployment, correctly handles a genuinely large range of real-world traffic — "moderate scale" is a much higher ceiling than it sounds like next to "web scale" marketing language
Right-sizing means estimating realistic traffic over a defined horizon (e.g. 12-18 months) with headroom, not building for an unbounded "what if we go viral" scenario that has no probability attached to it
The cost of over-engineering is not one-time — it recurs every month in infrastructure spend and every sprint in the engineering time needed to operate, monitor and reason about the extra moving parts
A cheaper design that can be evolved later (clean boundaries, a migration path already thought through) is usually better than an expensive design built for scale up front — premature scale-out is a form of premature optimization
This is not an argument against ever building for scale — it is an argument for building for the scale that is actually likely, with an explicit, cheap escape hatch for when growth exceeds the estimate

Same requirement, two architectures, very different ongoing cost

Same requirement, two architectures, very different ongoing cost
Dimension	Built for maximum hypothetical scale	Right-sized for realistic moderate scale
Database	Sharded across 12 nodes from day one	Single primary + read replica, vertically scaled
Deployment	Active-active across 3 regions	Single region, backups + a documented DR runbook
Monthly infra cost	High, paid continuously regardless of traffic	A fraction of the above, matched to actual load
Operational load	Cross-shard queries, multi-region consistency, more on-call surface	One system to reason about, fewer failure modes
Path if traffic exceeds estimate	Already built — no migration needed	A planned, budgeted migration triggered by a real signal

Two designs satisfying the same requirement

Built for maximum hypothetical scale

Sharded database, multi-region active-active from launch
Pays full infra cost every month regardless of traffic
Cross-shard queries and multi-region consistency bugs on-call has to own
No migration needed if growth ever arrives — but it may never arrive

Right-sized for realistic moderate scale

Single primary + read replica, single region
Infra cost matched to actual measured load
One system to reason about; fewer failure modes
A budgeted, planned migration path if a real growth signal appears

Command

text

// A right-sizing checklist before adopting a
// scale-out technique:
1. What is the realistic traffic in 12-18 months,
   not the best-case hypothetical?
2. Does the simpler, cheaper design fail at that
   realistic number, or only at 10-100x it?
3. What is the actual migration cost later, if
   growth exceeds the estimate -- is there a
   clean path, or is it a rewrite?
4. What does the complex design cost every month
   between now and the point growth might justify it?

Scale-out techniques (sharding, multi-region active-active, event-driven fan-out) cost money and operational complexity every month starting the day they ship — pay that cost when a measured growth signal justifies it, not against an unquantified "what if we go viral" scenario. Moderate scale sized to realistic traffic, with a documented migration trigger, satisfies the same requirement at a fraction of the cost.

Spotted a mistake on this page? Tell us.

Read Sheet