Monitoring — CloudWatch cheat sheet · AWS

1.Metrics, Dimensions, and Statistics

A CloudWatch metric is a time-ordered set of data points — one variable you watch over time, such as an EC2 instance's CPU usage. Each metric is identified by a namespace, a name, and up to thirty dimensions (name/value pairs like `InstanceId=i-0abc`). You never read the raw points directly: you ask for a statistic — Average, Sum, Maximum, a percentile — aggregated over a period you choose.

Key features

A metric is uniquely defined by a namespace, a name, and zero or more dimensions; every unique dimension combination is treated as a separate metric
You can assign up to 30 dimensions to a metric, and AWS namespaces follow the `AWS/{service}` convention (`AWS/EC2`, `AWS/Lambda`)
Standard resolution is one-minute granularity; high resolution is one second, and is only available on custom metrics you publish that way
A period is the aggregation window: valid values are 1, 5, 10, 30, or any multiple of 60 seconds, defaulting to 60
Retention is tiered — sub-60-second points for 3 hours, 60-second for 15 days, 5-minute for 63 days, 1-hour for 455 days — with automatic rollup as data ages
Metrics exist only in their own Region, cannot be deleted, and expire after 15 months with no new data; percentiles need raw, unsummarized points, so publishing a statistic set makes percentile statistics unavailable except in narrow cases

Metric retention and rollup

Metric retention and rollup
Period of the data points	Available for	What happens next
Under 60 seconds (high resolution)	3 hours	Rolled up into the 60-second series
60 seconds (1 minute)	15 days	Rolled up into 5-minute resolution
300 seconds (5 minutes)	63 days	Rolled up into 1-hour resolution
3600 seconds (1 hour)	455 days (15 months)	Expires on a rolling basis

What identifies a metric

AWS/ApplicationELB · TargetResponseTime · {LoadBalancer=app/api-lb, TargetGroup=tg-api} · p99 over 60s

AWS/ApplicationELB

Namespace — The container. Metrics in different namespaces never aggregate together.

TargetResponseTime

Metric name — The variable being measured over time.

{LoadBalancer=app/api-lb, TargetGroup=tg-api}

Dimensions — Name/value pairs that are part of the identity — a different combination is a different metric.

p99 over 60s

Statistic + period — How the points are aggregated, and over what window. Chosen at read time, not at publish time.

Signature

metric = namespace + name + dimensions[≤30] · read as statistic over period (1|5|10|30|60n s)

namespace + name + dimensions identify a metric; every dimension combination is its own metric, so keep cardinality bounded. Statistic and period are read-time choices. Retention is tiered with rollup — 1-minute data for 15 days, 5-minute for 63 days, 1-hour for 15 months.

2.Log Groups, Metric Filters, and Logs Insights

CloudWatch Logs stores log events. A log stream is a sequence of events from one source — one container, one function instance, one server. A log group is a set of streams that share retention, monitoring, and access-control settings, and it is the unit you actually configure. Metric filters turn matching log lines into a metric you can alarm on, and Logs Insights is the query language for reading the logs directly.

Key features

A log stream is a sequence of log events sharing the same source; a log group is a group of streams sharing retention, monitoring, and access-control settings, with no limit on how many streams it holds
By default log data is stored indefinitely — retention is a per-log-group setting you have to set, and unset means "never expire"
Expired events are marked for deletion and typically removed within 72 hours; once marked, they stop adding to storage cost
A metric filter matches a pattern in incoming log events and publishes a CloudWatch metric from it, which is how you alarm on "ERROR appeared in the log"
A subscription filter streams matching events in near real time to Lambda, Kinesis Data Streams, or Firehose — the route to a central log account or an external system
Access to log streams is controlled at the log group level, because streams are nested under groups

Reading logs: three different tools

Reading logs: three different tools
Tool	Use it when	What it produces
Metric filter	You want an alarm on something that only appears in logs	A CloudWatch metric, alarmable like any other
Logs Insights	You are investigating and do not know the query yet	Ad-hoc query results over a time range
Subscription filter	Events must reach another system as they arrive	A near-real-time stream to Lambda / Kinesis / Firehose

Log group as the unit of configuration

Log group /aws/ecs/api

stream: task/abc123

one container

stream: task/def456

another container

stream: task/ghi789

a third

What the group is wired into

Metric filter

pattern → metric → alarm

Subscription filter

Lambda / Kinesis / Firehose

Logs Insights

query across all streams

Signature

log group { retention, access, metric filters, subscriptions } ⊃ log streams (one per source)

Log stream = one source; log group = the shelf, and the unit for retention, access, filters, and subscriptions. Retention defaults to never expire — set it. Metric filters turn log patterns into alarmable metrics; Logs Insights is for investigation; subscription filters stream events onward.

3.Alarms, Dashboards, and Event-Driven Actions

An alarm watches one metric against a threshold over time and takes an action when the comparison holds. It has three states — OK, ALARM, and INSUFFICIENT_DATA — and it only acts on a sustained state change, not on merely being in a state. The action is a notification to an SNS topic or an Auto Scaling policy, which is what turns monitoring into something that does work rather than something someone has to be watching.

Key features

An alarm watches a single metric against a threshold over time, and invokes actions for sustained state changes only — being in a state is not enough, the state must have changed and held for the configured number of periods
The three states are OK, ALARM, and INSUFFICIENT_DATA; the last is a real state you must decide how to treat, not an error
Alarm actions are an SNS notification or an Auto Scaling policy; SNS is what fans that out to email, chat, a ticket system, or a Lambda
The alarm period must be at least the metric's resolution — 300 seconds for EC2 basic monitoring, 60 for detailed monitoring
Evaluation periods let you require N breaching data points before acting; CloudWatch notifies when the oldest point in the window is breaching and the others are breaching or missing
Custom metrics published with time stamps far from the current UTC time can leave alarms stuck in INSUFFICIENT_DATA or fire them late

Treat-missing-data choices, and what each one means

Treat-missing-data choices, and what each one means
Setting	Missing data is treated as	Fits when
notBreaching	Within the threshold	The metric is only published when something happens (e.g. an error count)
breaching	Breaching the threshold	Silence itself is the failure — a heartbeat metric
ignore	The alarm keeps its current state	Sparse data where you want the last real signal to persist
missing	INSUFFICIENT_DATA (the default behaviour)	You want the gap visible rather than interpreted

Alarm states

data arrives,within thresholddata arrives,breachingN evaluation periodsbreaching → actionrecovers →OK actionmetric stopsbeing publishedmetric stopsbeing published

INSUFFICIENT_DATA

start

OK

ALARM

Signature

alarm(metric, statistic, period, evaluation-periods, threshold, treat-missing-data) → SNS | Auto Scaling policy

An alarm watches one metric, has states OK / ALARM / INSUFFICIENT_DATA, and acts only on a sustained state change. Set the evaluation periods and the missing-data behaviour deliberately — those, not the threshold, are what make an alarm trustworthy. Dashboards explain; alarms detect.

4.Metrics vs Logs vs Traces vs Alarms

A metric is a number over time — cheap to store, aggregated, no detail about any one request. A log is a record of one event, with full detail and no aggregation. A trace follows one request across every service it touched. An alarm is not data at all: it is a rule over a metric that takes an action. Choosing the wrong one is the most common reason an investigation stalls.

Key features

Metrics aggregate: cost scales with the number of time series, not with traffic, so cardinality must stay bounded
Logs do not aggregate: cost scales with volume, and any identifier can appear in them because cardinality is free
A trace is a set of spans sharing a trace id, showing where a single request spent its time across services
An alarm is a rule with an action, evaluated over a metric — it produces no data of its own
A metric filter is the bridge from logs to metrics; structured JSON logging is what makes that bridge stable
The usual failure mode is asking one type to do another's job: high-cardinality identifiers in metrics, or per-request aggregation attempted by querying logs

Which telemetry type answers which question

Which telemetry type answers which question
Question	Type	Why not the others
Is the error rate above 1% right now?	Metric (+ alarm)	Counting log lines for this on every check is slow and expensive
Why did order 88214 fail?	Log	A metric has no room for an order id
Where did this slow request spend its 4 seconds?	Trace	Per-service logs show durations but not the causal chain
Should someone be woken up?	Alarm	Only alarms take action; the rest are data
How many unique users were affected?	Log query (Logs Insights)	User id as a metric dimension would create one metric per user

Aggregate or individual

Metrics + alarms

Aggregated numbers over time
Cheap at high traffic, expensive at high cardinality
Answer "is something wrong, and how much"
The only type an alarm can watch

Logs + traces

Individual events and individual requests
Cheap at high cardinality, expensive at high volume
Answer "what happened to this one request"
Where the detail an alarm cannot carry lives

Command

text

# The same incident, seen through each type
metric  Sum(ApiErrors) by minute            -> how bad, since when
alarm   Sum(ApiErrors) >= 5 for 2 periods    -> who gets woken up
trace   api 40ms | auth 15ms | db 3,900ms    -> where the time went
log     filter orderId = "88214"             -> what happened to one order

Metrics: aggregated numbers, bounded cardinality, what alarms watch. Logs: individual events, unlimited cardinality, where detail lives. Traces: one request across services. Alarms: a rule with an action, not a data type.

5.Designing Observability Around the Right Signals

Monitoring every metric a service emits produces noise. Pick a small set of signals that predict user pain — how slow requests are, how many fail, how close a resource is to its limit, how much work is arriving, how much is waiting — and alarm on those. Then add one or two business measures, because a system can be perfectly healthy by every technical measure while no orders are being placed.

Key features

Latency belongs at a percentile (p95, p99), not an average — the average hides the tail where the pain is
Error rate is a ratio, not a count: 50 errors means nothing without the request count underneath it
Saturation is how close a resource is to its limit — CPU, memory, connection pool, IOPS, concurrency — and it is the signal that predicts the other three going bad
Throughput is context rather than an alarm — a drop in requests is only an incident when it is unexpected for that time of day — while queue depth turns a downstream slowdown into a visible, growing number before it becomes a timeout
Database connections and cache hit rate are the two saturation signals that most often explain a latency graph nobody can account for
A business KPI (orders placed, signups completed) is the only signal that fails when everything technical is green and the system is still broken

A starting signal set, and what each one catches

A starting signal set, and what each one catches
Signal	Measured as	Catches
Latency	p95 / p99 per endpoint	Slowness the average hides
Error rate	5xx ÷ total requests	Failures, independent of traffic volume
Saturation	% of a hard limit (pool, CPU, concurrency)	The problem before it becomes latency
Throughput	Requests or messages per minute	Traffic loss, and context for the others
Queue depth	Messages visible / oldest message age	A consumer that cannot keep up
Cache hit rate	Hits ÷ (hits + misses)	A cache that silently stopped working

Signals in the order they usually move

Throughput up

more work arriving

Saturation up

pool, CPU, concurrency near limit

Queue depth up

work waiting, not yet failing

Latency up

p99 first, average later

Errors

timeouts — the user finally notices

Signature

latency(pNN) · error rate(ratio) · saturation(% of limit) · throughput · queue depth · cache hit rate · business KPI

Latency at a percentile, error rate as a ratio, saturation of the resource that actually binds you, throughput for context, queue depth as the early warning — plus at least one business KPI, because a system that receives no requests looks perfectly healthy.