beginnerintermediateadvanced

AWS · Section 39

Logging Architecture

Level: intermediate
Read: 25 min
Concepts: 3

Compute on AWS is replaced automatically — instances by an Auto Scaling group, tasks by a scheduler, Lambda environments by the platform — so a log file on the host is destroyed by normal operation, usually at the moment it becomes interesting. Centralized logging means writing to stdout or an agent and shipping every event off the host as it is produced, then querying a log group that spans the whole fleet. Beyond application logs, an account produces load balancer and CloudFront access logs (into S3, queried with Athena), CloudTrail (who called which API), VPC Flow Logs (which addresses talked, accepted or rejected — metadata, never payload, collected outside the traffic path), and per-service logs. Each has its own destination, its own retention setting, and its own bill; CloudWatch Logs retention defaults to never expire, and ingestion volume usually costs more than storage duration. Finally, a log group is a data store with the classification of whatever the application wrote into it, and it typically has far broader read access than the database holding the same fields — so keeping sensitive values out at the source, scoping read access at the group, and giving workload roles append-only permissions all matter as much as the pipeline itself.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

Automatic instance replacement and host-local log files are incompatible by design.
Application, access, CloudTrail, flow, and service logs each answer a different question and land in different places.
CloudWatch Logs retention defaults to never expire — every log group needs a deliberate policy.
Reducing what is emitted cuts a logging bill more than shortening retention does.
A log group carries the classification of its contents, and usually has wider read access than the database.

What you will be able to do

Design a logging path that survives instance and task replacement
Pick the right log source for a question, and know which ones land in S3 rather than CloudWatch Logs
Set retention deliberately and identify the largest driver of a logging bill
Emit structured JSON that a metric filter and Logs Insights can both work with
Scope log-group access, keep sensitive fields out at the source, and give workload roles append-only permissions

From an ephemeral host to a protected, queryable log estate

Off the instance, into a log group

Sources, structure, retention, cost

Access, redaction, encryption

Logging Architecture

Getting logs off ephemeral compute, knowing which source answers which question, and protecting the data that ends up in them.

Centralized Logging, Not Instance Files

coreintermediate

A log file on a server only exists while that server does. On AWS, instances are replaced by an Auto Scaling group, containers are stopped and rescheduled, and Lambda execution environments disappear — so the logs of the failure that caused the replacement disappear with it. Centralized logging means the application writes to standard output or to an agent, and something else ships those events off the host as they are produced.

Think of it as

Logs on the instance are notes written on the whiteboard of a room you are about to demolish. Centralized logging is the habit of photographing the whiteboard continuously, so the notes survive the demolition — which, in an Auto Scaling group, is scheduled rather than hypothetical.

What we're doing: Understand why an Auto Scaling group hides the very failures it is replacing instances for.

ephemeral-logs.txttext

03:14 Instance i-0abc starts failing its ELB health check. The reason is
      in /var/log/app/error.log on that instance.

03:16 The ASG marks it unhealthy, terminates it, and launches i-0def.

03:20 Someone is paged. i-0abc no longer exists; /var/log/app/error.log
      went with it. The dashboard shows a brief error spike and nothing
      that explains it.

1: The information needed exists, on disk, for about two minutes.
5: Instance replacement is the ASG working correctly. The logging design, not the scaling design, is what turned a self-healing event into an unexplainable one.

Why this works: Automatic replacement is the whole point of an Auto Scaling group, and it is fundamentally incompatible with logs that live on the instance. The two decisions have to be made together: anything that replaces compute automatically needs logs that leave the compute immediately.

Relying on SSH to read logs during an incident

Wrong

text

# ssh ec2-user@10.0.3.14 "tail -f /var/log/app/error.log"

Better

text

# Query the log group covering every task in the service, in Logs Insights

What you see: During an incident the on-call is guessing which of forty instances served the failing requests, and the one that did has already been replaced.

Why: Reading logs by host requires knowing which host, which is exactly what you do not know during an incident spread across a fleet. Centralized logs are indexed by time and service, which matches how the question is actually asked.

Where the log ends up

Local files

+Destroyed when the instance is replaced
+Readable only by someone who can log in
+One host at a time, no cross-fleet query
+Fills the disk, which itself causes outages

Centralized

−Survives the instance that produced it
−Read through IAM, not SSH
−One query across every task in the service
−Retention and cost are configured, not accidental

Remember: Compute on AWS is replaced automatically, so logs that live on it are destroyed by normal operation. Write to stdout or to an agent, ship every event off the host as it is produced, and read logs through a log group query rather than by logging into a machine.

AWS Log Sources, Structure, and Retention Cost

coreintermediate

An AWS account produces logs from several independent places, and each answers a different question. Application logs say what your code did. Load balancer access logs say what arrived and what status went back. CloudTrail says who called which API. VPC Flow Logs say which addresses and ports talked to each other. Each has its own destination, its own retention setting, and its own bill.

Think of it as

These are four different cameras pointed at the same building: one inside the office, one on the front door, one on the key-card system, one on the street. An incident review usually needs two of them, and which two depends on whether the problem was the code, the request, the permission, or the network.

What we're doing: Cut a CloudWatch Logs bill that has grown to be one of the largest lines on the account.

log-cost.txttext

Audit: 214 log groups. 61 have no retention policy at all.

Step 1 — set retention on every group. 30 days for application logs,
        400 days for audit-relevant ones, 7 days for build output.

Step 2 — turn off DEBUG in production. Ingestion volume drops 70%; this
        saves more than step 1, because ingestion is billed per GB in.

Step 3 — send high-volume, rarely-read logs (flow logs) to S3 instead of
        CloudWatch Logs, and query them with Athena.

3: The default is never expire, so a log group with no policy is not a decision anyone made — it is the absence of one.
6: Ingestion is charged on the way in and storage on what remains. Reducing what is emitted beats reducing how long it is kept.
9: S3 plus Athena costs less per GB than CloudWatch Logs for data you query occasionally rather than continuously.

Why this works: Log cost is driven by volume in, then by how long it stays. Teams usually reach for retention first because it is one setting, when the larger saving is in not emitting debug output from production at all — and in choosing S3 for the high-volume, rarely-read sources.

Logging unstructured text and parsing it with regular expressions later

Wrong

text

logger.info("user %s placed order %s for %.2f", user_id, order_id, total)

Better

text

logger.info(json.dumps({"event": "order_placed", "user_id": user_id,
                        "order_id": order_id, "total": total}))

What you see: Every Logs Insights query needs a bespoke `parse` pattern, and each one breaks the next time someone edits the message wording.

Why: A log line is read far more often by a query than by a human. Emitting JSON makes fields addressable by name in Logs Insights and matchable by a metric filter, which is what turns logs from something you read into something you can alarm on.

Four log families, four destinations

Into CloudWatch Logs

Application logs

stdout via agent or log driver

VPC Flow Logs

or S3 / Firehose

Service logs

Lambda, RDS, API Gateway

Into S3

ALB / CloudFront access logs

queried with Athena

CloudTrail trails

plus optional CloudWatch Logs

Which log answers which question

Which log answers which question
Question	Log source	Default destination
What did the code do while serving this request?	Application logs	CloudWatch Logs
Was the request even received, and what status went back?	ALB / CloudFront access logs	S3
Who called this AWS API, when, from where?	CloudTrail	S3 (+ CloudWatch Logs)
Did traffic reach the instance at all, or was it rejected?	VPC Flow Logs	CloudWatch Logs, S3, or Firehose
Why did this managed service misbehave?	Service logs (RDS, Lambda, API Gateway)	CloudWatch Logs

Together

text

# "Connection times out" — flow logs answer it before the app logs can
# REJECT means a security group or NACL refused it; no ACCEPT at all
# means the packet never arrived.
fields srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and action = "REJECT"
| stats count() by srcAddr

Remember: Application logs = what the code did. ALB/CloudFront access logs (in S3) = what arrived. CloudTrail = who called the API. VPC Flow Logs = which addresses talked, accept or reject, never payload. Set retention on every log group — the default is forever — and cut ingestion before you cut retention.

Protecting Logs and the Data In Them

coreintermediate

Logs accumulate whatever the application put in them — request bodies, headers, email addresses, sometimes tokens. That makes a log group a data store with the same classification as the data flowing through it, and it usually has far wider read access than the database does. Treat it accordingly: scope who can read it, keep sensitive fields out at the source, and encrypt what is kept.

Think of it as

A production database has a review process, an access request form, and an audit trail. The log group holding the same fields often has none of that, because it was created by a deploy script. The data does not care which one it is sitting in.

What we're doing: Stop treating a log group as less sensitive than the database it mirrors.

log-access-review.txttext

Who can read /aws/ecs/checkout today?
  Every engineer with ReadOnlyAccess — about 40 people.

Who can read the payments table in RDS today?
  Four people, via a reviewed access request.

The log group contains the order id, the user id, the email address, and
until last month the full request body. Same data, two very different
access policies — and only one of them was ever reviewed.

1: ReadOnlyAccess includes CloudWatch Logs. Broad read policies are usually granted with dashboards in mind, not log contents.
4: The database access path had a review because someone designed it. The log path had none because it was a side effect of deploying.
7: Retention makes this worse over time: data removed from the database last year may still be sitting in a log group set to never expire.

Why this works: The asymmetry is the point. Log access is granted broadly and early, database access narrowly and deliberately, and the two stores often hold the same fields. Reviewing who can read production log groups is usually the highest-value hour in a data-protection review.

Masking sensitive data at read time instead of keeping it out

Wrong

text

# Log the full request body, rely on a data protection policy to mask it

Better

text

# Do not log the body. Use masking as a safety net for what slips
# through, not as the primary control.

What you see: The raw value still exists in the stored event, so an export to S3, a subscription filter, or an unmask permission surfaces exactly the data the policy appeared to remove.

Why: Masking changes what a reader is shown, not what was written. Anything that copies the log data out — a subscription filter, an export, a cross-account share — starts from the stored event, so a value that was never emitted is the only one that is genuinely absent.

Where sensitive data gets into logs

Where sensitive data gets into logs
Source	Typical leak	Fix
Framework request logging	Full query string, including tokens in URLs	Log the route, not the raw URL; never put secrets in query strings
Exception handlers	A stack trace embedding the request body	Log an error code plus a request id; keep the body out
Debug logging left on	Whole objects, including PII fields	Turn debug off in production; log identifiers, not records
Third-party SDK logging	Authorization headers at DEBUG level	Pin the SDK's log level explicitly rather than inheriting root
Health-check verbosity	Internal hostnames and versions in every line	Reduce level; the volume also costs money

Together

text

# Log identifiers, not records
{"event": "payment_failed", "order_id": "88214",
 "user_id": "u_9931", "reason_code": "card_declined"}

# Not this
{"event": "payment_failed", "card": {"number": "4111...", "cvv": "123"},
 "user": {"email": "someone@example.com", "address": "..."}}

Remember: A log group is a data store with the classification of whatever the app logged, and usually far broader read access than the database. Keep sensitive fields out at the source, scope read access at the log group, encrypt with KMS where it matters, and never give a workload role more than append.