Logging Architecture

Centralized logging, the log sources an account produces, and protecting what ends up in them

Instance logs die with the instanceexactly when they become interesting
Four families, four destinationsapp, access, CloudTrail, flow logs
Retention defaults to never expireset it on every log group
A log group is a classified data storeusually with broader access than the database

1.Centralized Logging, Not Instance Files

A log file on a server only exists while that server does. On AWS, instances are replaced by an Auto Scaling group, containers are stopped and rescheduled, and Lambda execution environments disappear — so the logs of the failure that caused the replacement disappear with it. Centralized logging means the application writes to standard output or to an agent, and something else ships those events off the host as they are produced.

Key features

An unhealthy instance is terminated and replaced, so the local logs explaining why it was unhealthy are destroyed at exactly the moment they become interesting
A container writing to stdout/stderr has its output collected by the task's log driver (`awslogs` for CloudWatch Logs) with no file management in the container
Lambda writes to CloudWatch Logs by default; the execution environment is not something you can log into afterwards
A log group collects streams from every instance or task of a service, so a query can span the whole fleet instead of one host
Logging to a file and shipping it with an agent is still valid — the CloudWatch agent does this — but the file is a buffer, not the record
SSH-ing to a host to read logs does not scale past a handful of servers, and stops working entirely once instances are ephemeral

Where the log ends up

Local files

Destroyed when the instance is replaced
Readable only by someone who can log in
One host at a time, no cross-fleet query
Fills the disk, which itself causes outages

Centralized

Survives the instance that produced it
Read through IAM, not SSH
One query across every task in the service
Retention and cost are configured, not accidental

Signature

app → stdout / agent → log group (per service) → query across the fleet

Compute on AWS is replaced automatically, so logs that live on it are destroyed by normal operation. Write to stdout or to an agent, ship every event off the host as it is produced, and read logs through a log group query rather than by logging into a machine.

2.AWS Log Sources, Structure, and Retention Cost

An AWS account produces logs from several independent places, and each answers a different question. Application logs say what your code did. Load balancer access logs say what arrived and what status went back. CloudTrail says who called which API. VPC Flow Logs say which addresses and ports talked to each other. Each has its own destination, its own retention setting, and its own bill.

Key features

VPC Flow Logs capture information about IP traffic to and from network interfaces, publish to CloudWatch Logs, S3, or Amazon Data Firehose, and are collected outside the traffic path so they do not affect throughput or latency
Flow logs record metadata about a flow — addresses, ports, protocol, bytes, accept/reject — not packet payload, so they cannot show what was said
ALB and CloudFront access logs are delivered to S3, not CloudWatch Logs, which is why they are queried with Athena rather than Logs Insights
CloudWatch Logs retention is per log group and defaults to never expire, so every log group created without a retention policy accrues storage cost indefinitely
Structured JSON logging is what lets a metric filter match a named field and Logs Insights parse without regular expressions
Ingestion is usually the larger line on a logging bill than storage — dropping debug logging in production reduces cost more than shortening retention

Which log answers which question

Which log answers which question
Question	Log source	Default destination
What did the code do while serving this request?	Application logs	CloudWatch Logs
Was the request even received, and what status went back?	ALB / CloudFront access logs	S3
Who called this AWS API, when, from where?	CloudTrail	S3 (+ CloudWatch Logs)
Did traffic reach the instance at all, or was it rejected?	VPC Flow Logs	CloudWatch Logs, S3, or Firehose
Why did this managed service misbehave?	Service logs (RDS, Lambda, API Gateway)	CloudWatch Logs

Four log families, four destinations

Into CloudWatch Logs

Application logs

stdout via agent or log driver

VPC Flow Logs

or S3 / Firehose

Service logs

Lambda, RDS, API Gateway

Into S3

ALB / CloudFront access logs

queried with Athena

CloudTrail trails

plus optional CloudWatch Logs

Signature

app logs · ALB/CloudFront access logs (S3) · CloudTrail (S3) · VPC Flow Logs (CW/S3/Firehose) · service logs

Application logs = what the code did. ALB/CloudFront access logs (in S3) = what arrived. CloudTrail = who called the API. VPC Flow Logs = which addresses talked, accept or reject, never payload. Set retention on every log group — the default is forever — and cut ingestion before you cut retention.

3.Protecting Logs and the Data In Them

Logs accumulate whatever the application put in them — request bodies, headers, email addresses, sometimes tokens. That makes a log group a data store with the same classification as the data flowing through it, and it usually has far wider read access than the database does. Treat it accordingly: scope who can read it, keep sensitive fields out at the source, and encrypt what is kept.

Key features

Access to log streams is controlled at the log group level, because streams are nested under groups — there is no per-stream permission
A log group can be encrypted with a customer managed KMS key, which makes reading it require `kms:Decrypt` on that key as well as the Logs permission
CloudWatch Logs data protection policies can detect and mask sensitive data patterns in log events, so a masked value never appears to an ordinary reader
Redaction at the source is stronger than masking at read time: a value never written cannot leak through a subscription filter, an export, or a misconfigured policy
A subscription filter and an S3 export both copy log data out of the log group — every downstream destination inherits the same classification
Audit logs (CloudTrail) belong in a separate account with no path from ordinary workload roles, which is a stricter rule than for application logs

Where sensitive data gets into logs

Where sensitive data gets into logs
Source	Typical leak	Fix
Framework request logging	Full query string, including tokens in URLs	Log the route, not the raw URL; never put secrets in query strings
Exception handlers	A stack trace embedding the request body	Log an error code plus a request id; keep the body out
Debug logging left on	Whole objects, including PII fields	Turn debug off in production; log identifiers, not records
Third-party SDK logging	Authorization headers at DEBUG level	Pin the SDK's log level explicitly rather than inheriting root
Health-check verbosity	Internal hostnames and versions in every line	Reduce level; the volume also costs money

Signature

log group = classified data store · permissions at group level · redact at source · KMS for encryption at rest

A log group is a data store with the classification of whatever the app logged, and usually far broader read access than the database. Keep sensitive fields out at the source, scope read access at the log group, encrypt with KMS where it matters, and never give a workload role more than append.

Spotted a mistake on this page? Tell us.

Read Sheet