Monitoring — CloudWatch

What we're doing: Understand why a dashboard graph goes blank when you widen the time range past two weeks.

Widget: custom metric OrderLatency, statistic p99, period 60 seconds.

Last 6 hours   -> a line, one point per minute.
Last 7 days    -> still a line; 1-minute data is kept for 15 days.
Last 30 days   -> empty, because 1-minute data older than 15 days has been
                  rolled up and is only retrievable at 5-minute resolution.

Fix: set the widget's period to 300 for the 30-day view.

1

The period is a read-time choice, so the same stored metric can be graphed at several resolutions — but only at resolutions the data still exists at.

5

Nothing was deleted. The 1-minute series aged out of its 15-day window and the same data now lives at 5-minute granularity.

Why this works: CloudWatch rolls metric data up as it ages rather than keeping every point forever. A dashboard with a hard-coded short period looks broken on long time ranges, and the fix is to match the period to the age of the data rather than to assume the metric stopped being published.

What we're doing: Alarm on application errors that exist only as log lines, without shipping logs anywhere else.

Log group: /aws/ecs/api  (retention set to 30 days)

Metric filter: pattern { $.level = "ERROR" }
  -> metric ApiErrors in namespace Api/Prod, value 1 per match

Alarm: Sum(ApiErrors) >= 5 over 2 consecutive 1-minute periods
  -> SNS topic -> on-call

Nothing leaves CloudWatch Logs. The alarm fires on a pattern in the log
text itself, turned into a numeric series.

1

Retention is set on the group. Left alone it is "never expire", and the storage bill grows forever.

4

The filter pattern here is a JSON field match, which only works because the application logs structured JSON. Free-text matching is brittle by comparison.

7

Two consecutive periods, not one — a single blip should not page anyone. Sum, not Average, because the question is "how many errors", not "how error-ish".

Why this works: Metric filters are the bridge between the two data types: logs are where detail lives, metrics are what alarms work on. Structured JSON logging is what makes the bridge reliable, because the filter matches a named field instead of a substring that a message change can silently break.

What we're doing: Page on a real availability problem without paging on a single slow minute.

Metric: 5xx rate = Sum(HTTPCode_Target_5XX_Count) / Sum(RequestCount)
Period: 60s, evaluation periods: 3, datapoints to alarm: 2 of 3

Threshold: > 1%

Treat missing data: notBreaching — the metric is absent when no requests
arrive at 04:00, and no traffic is not an outage.

Action: SNS -> on-call rotation. OK action: SNS -> the same topic, so the
recovery is announced too.

2

2-of-3 tolerates one bad minute in a three-minute window. A 1-of-1 alarm on a noisy metric produces pages nobody trusts, which is worse than no alarm.

5

This is a design decision, not a default to accept. Getting it wrong is how an alarm pages at 4am every night, or never fires when the service is completely dead.

8

Without an OK action, the on-call has to check manually whether it recovered. The recovery notification is half the value of the alarm.

Why this works: The three knobs that decide whether an alarm is trusted are the statistic, the evaluation window, and the missing-data behaviour. Threshold is the one people tune, and it is usually not the one that is wrong.

What we're doing: Explain a latency incident that CPU and memory graphs showed nothing about.

14:02 DatabaseConnections on the RDS instance reaches its max_connections
      ceiling. CPU is 30%. Memory is fine.

14:03 Application connection pool: every checkout now waits. Queue depth
      on the request queue starts climbing.

14:05 p99 latency crosses 3s. Average latency is still 210ms, so the
      average-latency dashboard widget looks normal.

14:07 Gateway timeouts. Error rate alarm finally fires — five minutes
      after the saturation signal was already at 100%.

1

Saturation of a non-obvious resource. CPU and memory are the two most-watched signals and neither of them moved.

4

Queue depth is the earliest user-visible symptom, and it is a number, not an error — nothing has failed yet.

7

p99 moves minutes before the average does. An alarm on the average would have fired last, or not at all.

Why this works: Alarming only on errors means being told after users are affected. Saturation signals move first, which is what makes them worth the effort of instrumenting — connection pool usage and queue depth are usually the two that pay for themselves fastest.