AWS quick reference

unmanaged (EC2 you patch) vs. managed (RDS, SQS, Lambda)

A managed service takes an operational promise off your plate — ask what it guarantees, not just what it is called.

aws <service> <action> --region <region>

Every CLI/SDK call is Regional unless the service itself is global. Check which before assuming a resource "disappeared."

aws ec2 describe-instances --region eu-west-1

AWS: security OF the cloud · You: security IN the cloud

AWS secures infrastructure and (for managed services) the service internals. Configuration, access control, and data are always the customer's responsibility.

compute + storage + requests + data transfer + capacity

The five recurring dimensions nearly every AWS bill is built from. Read a new service's pricing page by mapping its charges onto these first.

Service Quotas console → search service → view/request increase

Where to check a limit before an architecture depends on being under it, and where to request a raise before production traffic hits it.

dev / staging / prod → separate AWS accounts

An AWS account is a hard security and billing boundary — split environments by account, not by naming convention.

SCP = permission ceiling, attached to an OU or account

An SCP restricts the maximum permissions available in an account, even to that account's own root user — it never grants anything by itself.

aws organizations attach-policy --policy-id p-abc123 --target-id ou-root-prod

dev · staging · prod · security-log-archive · shared-services

The minimal deliberate account set: one per lifecycle stage, plus a write-only logging account and a shared-services account for common tooling.

aws sts assume-role --role-arn <target-role-arn>

Temporarily become a role in another account. The role's trust policy in the target account decides who is allowed to assume it.

aws sts assume-role --role-arn arn:aws:iam::333333333333:role/DeployRole --role-session-name deploy

vend → attach SCPs → centralize logging → monitor for drift

The repeatable shape of standing up a new, compliant AWS account: automated creation, baseline guardrails attached at birth, and ongoing detective monitoring.

Region → Availability Zone → edge location

You choose the Region and often the AZ; AWS always routes to the nearest edge location — nobody chooses that one.

--multi-az · targets spread across ≥2 AZs

The baseline for any production workload: a managed database with Multi-AZ enabled, and load balancer targets spread across at least two Availability Zones.

aws rds create-db-instance ... --multi-az

Regional = isolated per Region · Global = one instance, whole account

A Region-wide event only ever affects that Region's Regional resources. A global service (IAM, Route 53, CloudFront) has no Regional fallback — every Region depends on the same instance of it.

RPO = data you can lose · RTO = time you can be down

State both before picking a DR strategy. Active/standby and active/active are two ways to hit a target RPO/RTO — not synonyms for "disaster recovery."

trace: compute → load balancer → data, per AZ

Blast-radius containment is verified by tracing one AZ failure through every layer and confirming each survives independently, with spare capacity to absorb the shift — not by one global setting.

Effect + Action + Resource + Condition

The four elements of every IAM policy statement. Default posture is deny — a statement exists to open access, narrowed by whichever elements are present.

{ "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:...", "Condition": {...} }

Identity-based · Resource-based (trust) · Boundary · Session

Five policy types, two jobs: identity-based and resource-based policies grant access; permissions boundaries and session policies only narrow what an existing grant can reach.

aws sts assume-role --role-arn ... --role-session-name ...

IAM users hold long-lived credentials; roles are assumed and hand back temporary ones (ASIA-prefixed, always expiring). Federation and IAM Identity Center let humans assume roles without ever creating an IAM user.

aws sts assume-role --role-arn arn:aws:iam::111122223333:role/deploy --role-session-name ci

default deny → explicit Allow needed → explicit Deny always wins

IAM evaluation in one line: nothing is allowed unless some policy explicitly allows it, and any explicit Deny anywhere applicable overrides every Allow.

arn:partition:service:region:account-id:resource

The fixed shape of every AWS resource identifier, and what a policy's Resource field actually matches. Use the policy simulator or Access Analyzer to troubleshoot AccessDenied, not trial-and-error grants.

arn:aws:s3:::reports-bucket/2026/*

aws accessanalyzer create-analyzer / list-findings

IAM Access Analyzer covers three jobs: external-access findings, unused-access analysis, and policy validation — created once, reviewed on a recurring schedule, not a one-time scan.

aws accessanalyzer list-findings --analyzer-arn arn:aws:access-analyzer:...

IAM Identity Center → permission sets → accounts

One central sign-in point (built-in or an external IdP) that assigns workforce users access across many AWS accounts via permission sets.

aws configure sso

AssumeRoleWithSAML · AssumeRoleWithWebIdentity

The two STS operations that exchange an external identity provider's token (SAML assertion or OIDC JWT) for temporary AWS credentials.

aws sts assume-role-with-web-identity --role-arn ... --web-identity-token $JWT

aws sts assume-role --role-arn <arn> --role-session-name <name>

Exchanges proof of identity for temporary credentials (ASIA-prefixed) that expire automatically, 15 minutes to 12 hours later.

aws sts assume-role --role-arn arn:aws:iam::123456789012:role/deploy --role-session-name ci

aws sts get-caller-identity

Reports which identity the credential provider chain actually resolved to — the fastest way to confirm before debugging a permissions error.

aws sts get-caller-identity

aws:MultiFactorAuthPresent

A policy Condition key that is true only inside a session authenticated with a verified MFA device — used to gate sensitive actions beyond sign-in MFA alone.

"Condition": { "Bool": { "aws:MultiFactorAuthPresent": "true" } }

aws <service> <op> --profile <name> --query <jmespath> --output <fmt>

The four flags that cover most day-to-day CLI use: which credentials, what region, what to extract, and how to print it.

aws s3 ls --profile prod --query "Contents[].Key" --output text

client.get_paginator(op).paginate(...) · client.get_waiter(name).wait(...)

The two SDK helpers that remove the two most commonly hand-rolled loops: paging through a list-style API and polling for a target state.

client.get_waiter("instance_running").wait(InstanceIds=[iid])

aws service-quotas get-service-quota --service-code <svc> --quota-code <code>

Reads the current numeric limit for a specific quota in the current Region — the starting point before assuming a failure is a permissions problem.

aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A

One-off → console. Repeated → script / CLI / IaC.

The rule of thumb for choosing between a console click and automation: repetition is the signal, not personal preference.

aws s3api create-bucket ... # in a committed script, not a console session

aws ec2 create-vpc --cidr-block <cidr>

Creates an isolated virtual network scoped to one Region and one IPv4 (and optionally IPv6) CIDR range — everything else (subnets, gateways, routes) is built inside it separately.

aws ec2 create-vpc --cidr-block 10.0.0.0/16

VPC → Subnets → Route tables → Gateways

The chain of components that together decide whether a resource can reach — or be reached from — outside its subnet.

aws ec2 create-subnet --vpc-id vpc-... --cidr-block 10.0.1.0/24 --availability-zone eu-west-1a

0.0.0.0/0 → igw-... = public · no such route = private

The one rule that decides public vs private: whether the subnet's route table sends default traffic to an internet gateway.

aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<id>

One shared CIDR allocation plan → distinct /16s per VPC

Coordinate CIDR ranges across teams/accounts from a single plan so any two VPCs can be peered later without re-addressing either one.

prod: 10.0.0.0/16 · staging: 10.1.0.0/16 · dev: 10.2.0.0/16

usable IPs = 2^(32 − prefix) − 5

AWS reserves 5 addresses in every subnet regardless of size — always plan from usable count, not the raw CIDR block size.

/24 → 251 usable · /28 → 11 usable (smallest AWS allows)

0.0.0.0/0 → igw-... (public) | nat-... (private, outbound-only) | absent (isolated)

The default route in a subnet's associated route table decides its internet reachability — the local route is automatic and always present.

aws ec2 create-route --route-table-id <id> --destination-cidr-block 0.0.0.0/0 --nat-gateway-id <nat-id>

NAT gateway = outbound-initiated only, never inbound

A structural property, not a configurable rule — a private subnet behind a NAT gateway can call out and get responses, but nothing can initiate a connection in.

app (private) → api.example.com ✓ · internet → app (private) ✗

Gateway endpoint (S3/DynamoDB, free) · Interface endpoint (most else, billed)

A private path to an AWS service that never touches a NAT gateway or the public internet — implemented as a route-table target or an ENI, depending on type.

aws ec2 create-vpc-endpoint --service-name com.amazonaws.<region>.s3 --vpc-endpoint-type Gateway

NAT gateway = hourly + per-GB, one per AZ (not per subnet)

The two independent cost drivers behind a NAT gateway bill, and why sharing one per AZ (not per subnet) is the standard resilient pattern.

3 AZs → 3 NAT gateways, shared by every private subnet within each AZ

Stateful · allow-only · attached to the resource (ENI)

A security group filters traffic at the resource level — return traffic for an allowed connection is automatically permitted, and there is no explicit deny rule.

aws ec2 authorize-security-group-ingress --group-id sg-... --protocol tcp --port 443 --source-group sg-alb

Stateless · subnet-level · first-matching-rule-number wins

A network ACL filters at the subnet boundary with numbered, ordered rules that can explicitly ALLOW or DENY — inbound and outbound are evaluated completely independently.

aws ec2 create-network-acl-entry --rule-number 100 --rule-action deny --cidr-block 203.0.113.0/24

Security group (most rules) → NACL (subnet deny) → WAF (content)

Reach for the simplest layer that can express the rule — most access control needs only a security group.

app tier: security group scoped to the load balancer's security group as source

Placement + security group + NACL — no single point of failure

Combine subnet placement, per-resource security groups, and per-subnet NACLs so one misconfiguration does not expose a resource by itself.

database: private subnet + scoped security group, not either alone

Reply → client's ephemeral port (1024–65535), not the service port

A NACL needs an explicit outbound rule for the ephemeral port range to let replies leave — a security group handles this automatically via statefulness.

Outbound NACL rule: ALLOW TCP 1024-65535 to 0.0.0.0/0

Alias record: apex-capable, free, auto-TTL A/AAAA record

Route 53's extension for pointing a domain (including the bare apex) at an AWS resource without the limitations a CNAME has.

example.com → ALIAS → my-alb-1234.eu-west-1.elb.amazonaws.com

Public hosted zone (internet) vs private hosted zone (associated VPCs only)

The same domain name can resolve differently depending on whether the query originates inside an associated VPC or from the public internet.

aws route53 create-hosted-zone --name internal.example.com --hosted-zone-config PrivateZone=true

Simple · Weighted · Latency-based · Failover · Geolocation · Multivalue

Route 53's routing policies each decide which of several records to answer with, based on a different signal — proportion, speed, health, or location.

{ "SetIdentifier": "canary", "Weight": 10, ... }

VPC Resolver at <CIDR base>+2 · outbound/inbound endpoints for hybrid DNS

Every VPC resolves public DNS and its own private hosted zones automatically — Resolver endpoints and forwarding rules extend that to on-premises DNS servers.

Forwarding rule: corp.internal. → 192.0.2.10:53 (on-prem DNS)

Peering (2 VPCs) · Transit Gateway (hub) · VPN (internet) · Direct Connect (dedicated)

Four ways to connect networks, at different scales — peering does not transit, so it stops scaling past a handful of VPCs.

aws ec2 create-vpc-peering-connection --vpc-id vpc-0a --peer-vpc-id vpc-0b

ALB = Layer 7 (HTTP-aware) · NLB = Layer 4 (TCP/UDP, static IP)

ALB routes by inspecting HTTP requests (path, host); NLB forwards connections at the transport layer with no HTTP visibility but extreme throughput and static IP support.

aws elbv2 create-load-balancer --type application # or --type network

Listener → Rules (priority order) → Target group → Targets

The chain that decides which specific, healthy target handles an incoming request — each target group tracks its own health independently.

aws elbv2 create-target-group --protocol HTTP --port 8080 --health-check-path /healthz

Any target serves any request → scale/replace targets freely

Statelessness means no per-target memory is load-bearing — the property that makes Auto Scaling and rolling deployments safe without client impact.

Session data in a shared store (Redis/DynamoDB), not in-process memory

ALB: cross-zone always on · NLB: off by default

Cross-zone load balancing lets any load balancer node send to any target regardless of AZ — check NLB's setting explicitly, since it defaults off.

aws elbv2 modify-load-balancer-attributes --attributes Key=load_balancing.cross_zone.enabled,Value=true

Internet → LB (public subnet) → targets (private subnet, sg-scoped)

The standard web architecture: the load balancer is the only internet-facing piece, and the app tier is reachable only from the load balancer's security group.

App tier security group source = ALB's security group, not 0.0.0.0/0

AMI (template) + instance type (CPU/mem/net) + user data (first-boot script)

The three pieces that define what an EC2 instance runs and how it configures itself the first time it boots.

aws ec2 run-instances --image-id ami-... --instance-type m6g.large --user-data file://init.sh

On-Demand · Savings Plans · Reserved Instances · Spot

Four ways to pay for EC2 compute, each trading commitment or interruption risk for a lower rate — match the option to the workload's actual shape.

Steady baseline → Savings Plan · Bursts → On-Demand · Batch → Spot

Vertical: bigger instance (ceiling, resize gap) · Horizontal: more instances (needs LB + stateless)

Two ways to add capacity — vertical is simpler but capped and requires a stop/start; horizontal scales further and tolerates single-instance failure.

m6g.large → m6g.2xlarge (vertical) vs 1x → 4x m6g.large behind an ALB (horizontal)

Cluster (latency) · Spread (max 7/AZ, isolation) · Partition (rack-aware, distributed systems)

Placement groups trade physical proximity (latency) against fault isolation — pick based on which the workload actually needs.

aws ec2 create-placement-group --strategy spread

CPU/network/disk I/O: default · Memory/disk space: needs CloudWatch agent

The gap between what CloudWatch reports on an EC2 instance by default and what requires the CloudWatch agent — missing data is not the same as a healthy reading.

Install the CloudWatch agent to get mem_used_percent and disk_used_percent

gp3 (default) · io2 (high IOPS) · st1/sc1 (large sequential/cold)

EBS volume types trade cost against IOPS and throughput characteristics — gp3 is the sensible starting default for most workloads.

aws ec2 create-volume --volume-type gp3 --size 100

Instance store: ephemeral, host-local · EBS: persistent, network-attached

Instance store data does not survive a stop or host failure — only use it for caches, scratch space, or data replicated elsewhere.

Database data → EBS. Cache/scratch → instance store is fine.

Fix → new AMI version → replace instances (never patch in place)

Immutable infrastructure treats an AMI as the single source of truth for what a healthy instance looks like — a fix updates the AMI, not a running instance.

EC2 Image Builder pipeline → ami-v2 → Auto Scaling launch template updated

Cattle, not pets — replace freely, never hand-tune in place

An instance that is too risky to replace is itself evidence of accumulated configuration drift — capture what makes it special in the AMI or launch config instead.

A manual fix that only exists via SSH history is drift, not a fix

min ≤ desired ≤ max · launch template · health check

The core Auto Scaling Group vocabulary — the three capacity numbers, what launches new instances, and what triggers a replacement.

aws autoscaling update-auto-scaling-group --min-size 2 --max-size 10 --desired-capacity 4

Target tracking (reactive) · Step (severity-mapped) · Scheduled (known-ahead)

Auto Scaling policy types answer different questions — hold a metric steady, respond to specific alarm severities, or scale ahead of a known pattern.

aws autoscaling put-scheduled-update-group-action --recurrence "0 8 * * *" --min-size 6

Attach target group → auto register/deregister · health check type: ELB for app-level signal

An ASG attached to a target group automates instance registration and can use the load balancer's own health check as its replacement trigger.

aws autoscaling attach-load-balancer-target-groups --target-group-arns <arn>

Any instance can be replaced at any time — design accordingly

Externalize state, handle SIGTERM within the deregistration delay window, and never depend on a specific instance ID persisting.

signal.signal(signal.SIGTERM, handle_sigterm)

cluster → service → task → task definition

A cluster groups services and tasks; a service maintains a desired count of tasks; each task runs one instance of a task definition (the versioned blueprint).

aws ecs create-service --cluster prod --task-definition web:3 --desired-count 3

--launch-type EC2 | --launch-type FARGATE

Same task definitions and services run on either capacity type — EC2 requires managing instances yourself, Fargate provisions compute per task automatically.

aws ecs create-service --cluster prod --task-definition web:3 --launch-type FARGATE

taskRoleArn (app permissions) vs executionRoleArn (ECS startup permissions)

The task role is vended to the running application; the execution role is used by ECS itself to pull images, fetch secrets, and write logs before the app starts.

{ "taskRoleArn": "...", "executionRoleArn": "..." }

minimumHealthyPercent / maximumPercent

Bound how far a rolling ECS deployment can shrink or grow the running task count while replacing tasks with a new revision.

{ "deploymentConfiguration": { "minimumHealthyPercent": 100, "maximumPercent": 200 } }

ECS vs EKS

Choose ECS for simpler AWS-native container orchestration; choose EKS when Kubernetes portability or its ecosystem is the actual requirement.

aws ecs create-service --launch-type FARGATE # vs eksctl create cluster

Control plane always managed · nodes managed only in Auto Mode

EKS always runs the Kubernetes control plane for you; standard mode still requires you to manage nodes, while Auto Mode extends AWS management to nodes as well.

eksctl create cluster --name prod --enable-auto-mode

pod → deployment → service → ingress

Pods are the deployable unit; deployments keep replica counts running; services give pods a stable identity; ingress routes external traffic in. EKS Pod Identity (preferred over IRSA) grants pods scoped IAM permissions via their service account.

kubectl get pods,deployments,services,ingress -n production

EKS vs ECS decision

EKS is justified when Kubernetes portability, an existing investment, or its ecosystem is the actual requirement; otherwise ECS is usually the simpler AWS-native choice.

# EKS: reuse existing Helm charts and Kubernetes expertise across environments

Managed control plane ≠ no Kubernetes expertise needed

EKS removes control-plane (and optionally node) operations, but version upgrades, networking/ingress, and workload tuning remain real, ongoing Kubernetes-specific work.

# Plan Kubernetes version upgrades even on EKS Auto Mode

Init (once) → Invoke (per request, timeout-bound) → Shutdown

Code outside the handler runs once per execution environment during Init; handler code runs per request during Invoke. Versions are immutable; aliases are mutable pointers callers target.

s3 = boto3.client("s3") # Init — outside the handler, reused across invokes

sync (waits) · async (queues, returns 202) · event source mapping (Lambda polls)

Synchronous invocation blocks the caller; asynchronous invocation queues the event and returns immediately; an event source mapping has Lambda itself poll a source like SQS or Kinesis.

aws lambda invoke --invocation-type Event --function-name my-fn --payload '{}' out.json

reserved concurrency (capacity cap) vs provisioned concurrency (pre-warmed)

Reserved concurrency sets a min/max on concurrent executions without pre-warming anything. Provisioned concurrency pre-initializes environments to eliminate cold starts, at extra cost.

aws lambda put-provisioned-concurrency-config --function-name my-fn --qualifier LIVE --provisioned-concurrent-executions 20

At-least-once delivery → idempotent handlers

Lambda's async retries and event-source polling can redeliver the same event — handlers should be idempotent, commonly via a conditional write keyed on a unique event ID.

# DynamoDB conditional put keyed on event_id to detect and skip duplicates

Lambda vs long-running service

Lambda fits bursty, event-driven, short-lived work with zero idle cost; a long-running service fits steady high throughput, workloads over 15 minutes, or specialized networking.

# S3 upload → Lambda thumbnail; 24/7 transcoding → ECS

REST API (full features) vs HTTP API (minimal, cheaper)

REST APIs support API keys, usage plans, request validation, and WAF; HTTP APIs are a cheaper subset with JWT authorizers and automatic deployments but none of those REST-only features.

aws apigateway create-deployment --rest-api-id abc123 --stage-name prod

VPC link: API Gateway → private ALB/NLB/Cloud Map

A VPC link routes API Gateway traffic to a backend inside a VPC without the backend needing a public IP — supported targets are NLB/ALB on REST APIs, plus Cloud Map on HTTP APIs.

aws apigatewayv2 create-vpc-link --name backend-link --subnet-ids subnet-1a subnet-1b

29s integration timeout · 10 MB payload (REST APIs)

API Gateway REST APIs cap integration timeout at 29 seconds and payload size at 10 MB, independent of the backend's own configured limits — longer or larger workloads need an async pattern.

# Long-running work: return 202 immediately, notify/poll for completion instead of waiting synchronously

ALB vs API Gateway

Choose ALB for pure HTTP routing/load balancing to a running service without API-management needs; choose API Gateway when request validation, usage plans, API keys, or Lambda-native integration are required.

# 60s+ requests, no API keys needed → ALB, not API Gateway

bucket → key (object) — no real folders, only shared prefixes

An S3 object is addressed by a unique key inside a bucket; "folders" are a UI convention over a flat namespace where keys share a prefix.

aws s3api put-object --bucket my-bucket --key photos/2026/trip.jpg --body trip.jpg

Strong consistency (object PUT/DELETE) ≠ durability ≠ availability

S3 object writes are strongly consistent everywhere; durability (surviving over time) and availability (reachable now) are separate, independent guarantees — and S3 is not a POSIX filesystem.

# read-modify-PUT the whole object — there is no in-place append or partial write

Multipart upload: 5 MiB–5 GiB parts, ≤10,000 parts, ≤48.8 TiB object

Multipart upload splits a large object into independently-retriable parts, recommended once an object nears 100 MB. Range GET requests fetch a byte range of an object without downloading it whole.

aws s3api upload-part --bucket my-bucket --key large-file.zip --part-number 1 --upload-id <id> --body part1

Block Public Access: override, on by default

S3 Block Public Access overrides any bucket policy or ACL that would grant public access — keep all four settings enabled unless a specific use case requires public objects.

aws s3api get-public-access-block --bucket my-bucket

Static assets · uploads · backup/archival · data lake

The same S3 primitives (storage classes, lifecycle rules, presigned URLs, event notifications) combine into different shapes for static asset delivery, direct client uploads, backup/archival, and data lake storage.

aws s3 presign s3://my-bucket/uploads/file.jpg --expires-in 300

Identity-based (on the user/role) vs bucket policy (on the resource)

Identity-based policies control what an IAM identity can attempt; bucket policies are resource-based and are the standard way to grant another AWS account direct access to a bucket.

{ "Principal": { "AWS": "arn:aws:iam::222233334444:root" }, "Action": "s3:GetObject" }

SSE-S3 (default) · SSE-KMS (auditable, extra grant needed) · client-side

SSE-S3 is the automatic free default; SSE-KMS adds auditability and rotation but needs a separate kms:Decrypt grant and, for cross-account sharing, a customer-managed key; client-side encryption keeps plaintext away from AWS entirely.

--server-side-encryption aws:kms --ssekms-key-id alias/my-key

Governance (bypassable) vs compliance (no override, ever) mode

Object Lock requires versioning. Governance mode can be bypassed with a special IAM permission; compliance mode cannot be overridden by anyone, including root, until the retention period expires.

aws s3api put-object-retention --bucket my-bucket --key file.txt --retention Mode=COMPLIANCE,RetainUntilDate=2027-01-01T00:00:00Z

Live replication (forward-only) vs S3 Batch Replication (existing objects)

Live CRR/SRR replicates new and updated objects going forward only; S3 Batch Replication handles pre-existing, previously-failed, or already-replicated objects on demand.

# Batch Replication job needed to backfill objects that predate a new CRR rule

Least privilege default — public is a reviewed exception

Scope S3 access as narrowly as the requirement allows (presigned URLs, access points, scoped bucket policies); treat a public bucket as a deliberate, reviewed exception, usually fronted by CloudFront.

# User-specific file access → presigned URL, not a public bucket

Distribution → cache behavior → cache policy + origin request policy

A distribution routes requests via path-matched cache behaviors, each with a cache policy (what varies the cache key) and an origin request policy (what is forwarded to the origin on a miss).

# Narrow cache key: only the headers/cookies/query strings that actually change the response

S3 + OAC (static) vs ALB/API Gateway (dynamic)

CloudFront + S3 with Origin Access Control fits static assets, keeping the bucket private; CloudFront + ALB/API Gateway fits dynamic content, still providing TLS termination and DDoS absorption regardless of cache hit rate.

Cache behavior "/static/*" → S3 (OAC); "/api/*" → ALB

Invalidation (costs money, incomplete) vs versioned filenames (free, complete)

TTL defaults to 24 hours; invalidation forces a manual CloudFront-only cache clear at a cost, while versioned filenames guarantee a cache miss everywhere for free — AWS's recommended approach for frequently-updated content.

aws cloudfront create-invalidation --distribution-id ABCDEF --paths "/images/*"

Automated backups (continuous, windowed) vs manual snapshot (deliberate, independent)

RDS automated backups enable point-in-time recovery within a retention window; a manual snapshot is explicitly triggered and retained independently — take one before risky changes like migrations.

aws rds create-db-snapshot --db-instance-identifier prod-db --db-snapshot-identifier pre-migration-snapshot

Multi-AZ (availability) vs read replicas (read scaling)

A single-standby Multi-AZ deployment provides synchronous automatic failover but never serves reads; read replicas are asynchronous copies built for read scaling and are not automatically promoted on failure.

aws rds create-db-instance-read-replica --db-instance-identifier read-1 --source-db-instance-identifier prod-db

Connection exhaustion (pooling fix) vs slow query (indexing fix)

A DB instance has a hard connection limit — pooling addresses exhaustion. Performance Insights and Enhanced Monitoring surface slow queries and OS metrics, but index design and query tuning remain the customer's responsibility.

# Add RDS Proxy or PgBouncer instead of repeatedly raising max_connections

Reconnect via hostname, retry with backoff — never cache the IP

RDS failover works by re-pointing the DB endpoint's DNS to a new IP — applications must reconnect via the hostname and retry transient failures with backoff rather than caching a resolved IP indefinitely.

for attempt in range(5): try: connect(); break; except: sleep(2**attempt)

Cluster endpoint (writer) · reader endpoint (readers) · shared cluster volume

Aurora separates compute from storage — every instance shares one cluster volume. The cluster endpoint always follows the current writer; the reader endpoint load-balances across readers, both automatically re-pointing on failover.

aws rds describe-db-clusters --query '*[].{Endpoint:Endpoint,ReaderEndpoint:ReaderEndpoint}'

Aurora PostgreSQL-Compatible vs Aurora MySQL-Compatible

Two separate Aurora offerings, each compatible with a specific upstream engine version — chosen at cluster creation; switching later is a full cross-engine migration.

# Existing PostGIS/pg_trgm usage → Aurora PostgreSQL-Compatible, not MySQL

Aurora (read scaling, auto-scaling storage) vs standard RDS (simpler, other engines)

Aurora fits PostgreSQL/MySQL workloads needing many low-lag readers, storage auto-scaling, or fast failover; standard RDS is simpler/cheaper for steady workloads or engines Aurora doesn't support (SQL Server, Oracle, Db2).

# SQL Server requirement rules out Aurora entirely — standard RDS only

Query (key-based, cheap) vs Scan (full read, expensive) · GSI (own key, anytime) vs LSI (shared key, at creation)

Query reads efficiently via the key structure; Scan reads the whole table before filtering. A GSI has its own key and can be added anytime; an LSI shares the base partition key and must exist at table creation.

aws dynamodb query --table-name Music --key-condition-expression "Artist = :a"

Single-table design: known access patterns → one table, prefixed keys, few GSIs

Single-table design stores multiple entity types in one DynamoDB table via generic prefixed keys, resolving known access patterns in one Query — it requires those patterns to be known upfront, unlike a relational database's ad hoc query flexibility.

PK: USER#123, SK: ORDER#456 — fetches a user and their orders in one Query

3,000 RU / 1,000 WU per partition per second — a hard ceiling

Every DynamoDB partition caps at 3,000 read units and 1,000 write units per second regardless of table-level capacity — a low-cardinality or skewed partition key creates a hot partition; adaptive capacity mitigates but does not replace good key design.

# Prefer a naturally high-cardinality partition key (user ID) over a low-cardinality one (status)

ConditionExpression (cheap, single-item) vs TransactWriteItems (2x cost, multi-item all-or-nothing)

A ConditionExpression on a plain write gives single-item optimistic concurrency cheaply; TransactWriteItems/TransactGetItems provide all-or-nothing, serializable multi-item operations but cost double capacity per item, even on cancellation.

UpdateItem with ConditionExpression "version = :expectedVersion"

Memcached (simple, multithreaded, no failover) vs Valkey/Redis OSS (data structures, pub/sub, failover)

Memcached fits pure key-value object caching scaled out across nodes. Valkey/Redis OSS fits workloads needing sorted sets/hashes/lists, pub/sub messaging, or replication with automatic failover.

aws elasticache describe-cache-engine-versions --engine memcached

Cache-aside (read-fill, can stale) vs write-through (write-fill, never stale) + TTL (staleness bound)

Cache-aside populates the cache on a read miss and costs a 3-trip miss penalty; write-through populates it on every database write and is never stale but leaves gaps on a fresh node. TTL expires keys to bound staleness in either strategy.

cache.set(key, value, ttl=300) # 5-minute staleness bound

SET key value NX PX ttl — acquire; check-then-DEL by token — release

A cache stampede is many concurrent requests missing a hot key at once. A SET NX PX lock lets one process rebuild it, but is best-effort — a paused holder can still lose the lock to a second acquirer before finishing.

SET lock:key token NX PX 5000

SET ... EX (sessions) · INCR + EXPIRE (rate limits/counters) · PUBLISH/SUBSCRIBE (pub/sub)

Session storage and rate limiting both use a TTL-bearing key; counters and rate limits both use atomic INCR/DECR; pub/sub delivers to current channel subscribers only and is unrelated to the key space.

INCR ratelimit:user42:1200 ; EXPIRE ratelimit:user42:1200 60

Multi-AZ: promote least-lagging replica → same primary endpoint via DNS → typically seconds

Multi-AZ on Valkey/Redis OSS promotes the replica with the least replication lag when the primary fails, resuming writes in typically a few seconds via the same re-pointed primary endpoint — asynchronous replication means a small amount of very recent data can still be lost. Memcached has no failover.

aws elasticache modify-replication-group --replication-group-id my-group --automatic-failover-enabled --multi-az-enabled --apply-immediately

Cache = expendable, rebuildable copy. System of record = durable store (or Valkey durability, deliberately enabled).

Cached data should always be treated as stale and safe to lose — a node replacement, eviction, or failover can destroy anything stored only in the cache. Use a durable data store as the real copy, or explicitly enable Valkey durability for a genuine system-of-record use case.

db.insert(id, data); cache.set(id, data, ttl=600) # durable copy first, cache is the accelerator

Visibility timeout (30s default, 12h max) · Long polling (WaitTimeSeconds up to 20s) · Standard (unlimited, best-effort order) vs FIFO (300 TPS default, strict order per group)

A received message is hidden for the visibility timeout, then reappears if not deleted. Long polling waits for a message instead of returning empty. Standard queues favor throughput; FIFO queues favor ordering within a message group.

aws sqs receive-message --queue-url $URL --wait-time-seconds 20 --visibility-timeout 60

Redrive policy: maxReceiveCount (1-1,000) · DLQ type must match source · DLQ redrive moves messages back

A message moves to the DLQ after maxReceiveCount failed receives. The DLQ must be the same queue type as its source and needs its own longer retention. Because delivery is at-least-once, consumers must be idempotent independent of the DLQ.

aws sqs set-queue-attributes --queue-url $URL --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"}'

At-least-once (standard) · exactly-once within dedup window (FIFO) · idempotency key + conditional write

A message may be delivered more than once; the consumer, not the queue, is responsible for tolerating that. An idempotency key checked via a conditional write is the standard pattern.

PutItem(table, key=message_id, ConditionExpression="attribute_not_exists(id)")

ChangeMessageVisibility --visibility-timeout <seconds> · 12h ceiling from first receive, never reset

Extend an in-flight message's visibility timeout with ChangeMessageVisibility, ideally via a periodic heartbeat for variable-length work. The 12-hour cap is absolute regardless of extensions.

aws sqs change-message-visibility --queue-url $URL --receipt-handle $RH --visibility-timeout 120

Lambda SQS maximum concurrency: 2-1,000 · default scaling caps at 1,250 · FIFO capped by MessageGroupId count too

Cap concurrent consumers below what a downstream dependency can sustain using maximum concurrency (Lambda) or a fixed worker pool size. FIFO throughput is also capped by the number of distinct message groups.

aws lambda update-event-source-mapping --uuid $UUID --scaling-config '{"MaximumConcurrency":50}'

Topic → N subscriptions (SQS | Lambda | HTTP(S) | email | SMS | push | Firehose), each with an optional filter policy and optional DLQ

A publish reaches every subscription whose filter policy matches (or every subscription with no filter policy). Each subscription retries failed server-side deliveries on its protocol's own schedule, then discards unless a dead-letter queue is attached.

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:111122223333:orders --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:111122223333:paging-queue

SNS topic --subscribe--> SQS queue (× N consumers), each queue polled independently

Each SQS queue subscribed to an SNS topic receives its own full copy of every published message. One queue per consumer gives fan-out; consumers sharing one queue instead split the message stream.

aws sns subscribe --topic-arn $TOPIC_ARN --protocol sqs --notification-endpoint $QUEUE_ARN

Direct call: caller blocks, must know callee. Publish: caller returns immediately, subscribers unknown to caller.

A synchronous call fits when the caller needs a response to continue. Pub/sub fits when the caller does not need a response, or multiple services must react to one event without the publisher knowing who they are.

sns.publish(TopicArn=ORDER_PLACED_TOPIC, Message=json.dumps(order))

bus → rule (pattern | schedule) → up to 5 targets, parallel

An event bus routes events; a rule matches by pattern or schedule and fans out to targets in parallel; an archive stores matches for replay to the same source bus.

aws events put-rule --name order-placed --event-pattern file://pattern.json

event: "XHappened" (broadcast, N listeners) vs command: "DoX" (one handler, expects an outcome)

A domain event announces something that already happened, with no known subscriber count. A command instructs one specific handler to perform an action and expects a result.

{ "detail-type": "OrderPlaced" } // event, not { "detail-type": "PlaceOrder" } // command

versioning: additive fields · duplicates: dedup by event id · retries: RetryPolicy → DLQ · consistency: eventual, not immediate

Loosely coupled event-driven services must handle changing event shapes, at-least-once (duplicate) delivery, automatic retries that exhaust to a dead-letter queue, and a subscriber's data lagging the source event.

if already_processed(event["id"]): return

Stream = set of shards · partition key → MD5 hash → shard · order guaranteed per-shard only

A record's partition key is hashed to pick its shard. Records on the same shard stay in write order. Retention defaults to 24 hours, raisable to 365 days, which is what enables replay.

aws kinesis put-record --stream-name orders --partition-key user-42 --data "..."

Shard: 1,000 rec/s or 1 MB/s write · 2 MB/s read (shared or, per enhanced fan-out consumer, dedicated)

Shared consumers poll GetRecords and split a shard's 2 MB/sec read throughput. Enhanced fan-out consumers each get their own 2 MB/sec, pushed via SubscribeToShard. Over-limit calls throw ProvisionedThroughputExceededException; splitting a hot shard adds capacity.

aws kinesis update-shard-count --stream-name orders --target-shard-count 4 --scaling-type UNIFORM_SCALING

Kinesis: ordered, multi-consumer, replayable · SQS: single-worker work queue · EventBridge: content-based event routing

Choose Kinesis when multiple independent consumers must read the same ordered, replayable data. Choose SQS for a work queue processed once per message. Choose EventBridge to route typed events to targets by rule.

Fulfillment worker pool → SQS · order.created fan-out → EventBridge · clickstream to two real-time consumers → Kinesis

Producers → Kinesis stream → {Flink (aggregate), Firehose (deliver to S3/Redshift/OpenSearch), Lambda (custom logic)}

One ingestion stream feeds several independent, concurrent consumers, each doing a different job on the same ordered data — real-time aggregation, data lake delivery, and custom alerting logic all draw from the same source.

A clickstream feeding a live dashboard (Flink) and a data lake (Firehose to S3) from the same Kinesis stream

StartAt, States, Task { Resource, Next | End }

A state machine is a named-state JSON workflow (ASL); a Task state runs one unit of work. Each run is an execution; Step Functions records every state transition as execution history.

"ValidateOrder": { "Type": "Task", "Resource": "arn:...:function:ValidateOrder", "Next": "ChargeCard" }

Retry { ErrorEquals, IntervalSeconds=1, MaxAttempts=3, BackoffRate=2.0 } · Catch { ErrorEquals, Next } · Parallel vs Map · TimeoutSeconds/HeartbeatSeconds · .waitForTaskToken

Retry re-runs the same state on matched errors with a growing wait; Catch reroutes once retries are exhausted. Parallel runs a fixed set of branches; Map runs one branch per array item. TimeoutSeconds/HeartbeatSeconds bound Task duration; .waitForTaskToken pauses for an external SendTaskSuccess/SendTaskFailure call.

"Retry": [ { "ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 3, "BackoffRate": 2.0 } ]

Orchestration in one Lambda (hidden, 15-min timeout) vs Step Functions (declarative, inspectable, up to 1 year)

Hand-written retry/catch logic across several service calls inside one Lambda hides the workflow in code and hits a 15-minute function timeout. Step Functions makes each step, retry, and failure a declarative, inspectable part of a state machine, running up to one year (Standard).

# Lambda growing its own try/catch + retry loop around 3+ service calls -> move to a state machine

Protect (KMS/Secrets Manager/Parameter Store) · Stop (WAF/Shield) · Detect (GuardDuty/Inspector/Macie) · Audit (Security Hub/Detective/CloudTrail/Config/IAM Access Analyzer)

Thirteen AWS security services sorted into four questions: what protects a secret, what stops an attack in-flight, what detects a threat or weakness, and what investigates or audits activity after the fact.

A public S3 bucket with PII: Macie finds the data, GuardDuty flags the exposure, Security Hub aggregates both, Detective and CloudTrail explain how it happened.

Key policy (required, one per key) · Grant (optional, temporary, layered on top)

A KMS key always has exactly one key policy. Grants add narrow, temporary permissions without editing it. Encryption context authenticates an encrypt/decrypt call without being secret. Customer managed keys are the only type that supports cross-account sharing and optional rotation; AWS managed keys rotate automatically every year with no opt-out.

aws kms create-grant --key-id alias/my-key --grantee-principal arn:aws:iam::111122223333:role/my-role --operations Decrypt

At rest: server-side encryption (KMS-backed) · In transit: TLS (ACM certificates)

Encryption at rest protects data in storage; encryption in transit protects data on the network. Independent controls — enabling one does not enable the other, so both need explicit verification.

aws s3api put-bucket-policy --bucket my-bucket --policy '{"Statement":[{"Effect":"Deny","Principal":"*","Action":"s3:*","Condition":{"Bool":{"aws:SecureTransport":"false"}}}]}'

Key policy (required, must enable IAM) + IAM policy (only matters once enabled)

A KMS key always evaluates its own key policy first. IAM policies can only grant access once the key policy explicitly enables them — an IAM Allow is not sufficient on its own, though an IAM Deny always applies.

secretsmanager:GetSecretValue(SecretId) · ssm:GetParameter(Name, WithDecryption=True)

Secrets Manager for credentials with automatic rotation; Parameter Store for general config, optionally KMS-encrypted as SecureString, no rotation built in.

aws secretsmanager get-secret-value --secret-id prod/orders-db/password

Management events (default) vs data events (opt-in, billed) · Event history (90 days) vs trail (durable)

Management events cover control-plane actions and are logged by default; data events cover high-volume resource-level actions and must be turned on. Event history is a free 90-day window; a trail is what you configure for durable, S3-delivered retention.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteObject

Organization trail → dedicated log archive account → least-privilege bucket policy + log file integrity validation

Centralize every account's CloudTrail events into one bucket in a dedicated log archive account that workload roles have no access to, and enable log file integrity validation so any tampering is cryptographically provable.

aws cloudtrail create-trail --name org-trail --s3-bucket-name log-archive-bucket --is-organization-trail --enable-log-file-validation

who: userIdentity.arn · what: eventSource+eventName · when: eventTime · where: sourceIPAddress · as-whom: sessionIssuer/sourceIdentity

The five audit answers CloudTrail's event record fields map to directly — no cross-system correlation required to read who did what, when, from where, and under which assumed identity.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=jordan.reyes

recorder → configuration item → configuration history · snapshot (all resources, one moment) · stream (SNS)

The configuration recorder writes a configuration item per change; items accumulate into a per-resource configuration history in S3. A snapshot captures every recorded resource at once; the configuration stream pushes each item to SNS.

aws configservice get-resource-config-history --resource-type AWS::EC2::SecurityGroup --resource-id sg-0a1b2c3d

Config rule (managed | custom Lambda/Guard) → COMPLIANT / NON_COMPLIANT / ERROR / NOT_APPLICABLE → remediation

Rules check resource configuration on change or on a schedule; detective mode covers deployed resources, proactive mode checks proposed properties without blocking. Conformance packs bundle rules and remediation; aggregators centralize the results read-only.

aws configservice describe-compliance-by-config-rule --compliance-types NON_COMPLIANT

CloudTrail → API activity (who/what/when) · AWS Config → resource configuration state (what it looked like)

Use CloudTrail to attribute a change to a principal and AWS Config to see the configuration before and after it. Both are regional and both need organization-level setup (an organization trail, a Config aggregator).

Config: get-resource-config-history · CloudTrail: lookup-events --lookup-attributes AttributeKey=ResourceName,...

web ACL { rules[priority] → Allow | Block | Count | CAPTCHA | Challenge, default action } → protected resource

Rules inspect requests and take an action; Allow and Block terminate evaluation, Count does not. Rule groups (AWS Managed Rules, Marketplace, or your own) make rules reusable. Scope is CLOUDFRONT (us-east-1 only) or REGIONAL.

aws wafv2 create-web-acl --scope=CLOUDFRONT --region=us-east-1 --default-action Allow={} --rules file://rules.json

IP set (known addresses) · rate-based rule (volume per key) · Bot Control (labels) · logs → CloudWatch Logs | S3 | Firehose

Use an IP set for known addresses, a rate-based rule for volume (keyed on IP, header, cookie, or a combination), and Bot Control labels for automated traffic. WAF logging is the only record of a blocked request.

RateBasedStatement { Limit: 2000, AggregateKeyType: CUSTOM_KEYS, CustomKeys: [{ Header: { Name: "x-api-key" } }] }

Shield Standard (automatic, free, L3/L4) · Shield Advanced (subscription, deeper L7 + response team) · WAF shapes L7

Shield absorbs volumetric and protocol floods; Shield Advanced adds deeper application-layer mitigation and the AWS Shield Response Team. A layer 7 request flood is shaped by AWS WAF rate-based rules at the edge, not by Shield alone.

aws shield describe-subscription · aws shield list-protections

metric = namespace + name + dimensions[≤30] · read as statistic over period (1|5|10|30|60n s)

Each unique dimension combination is a separate metric. Statistic (Average, Sum, Max, pNN) and period are chosen at read time. Retention is tiered with automatic rollup: 1-minute for 15 days, 5-minute for 63 days, 1-hour for 15 months.

aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name TargetResponseTime --period 60 --extended-statistics p99

log group { retention, access, metric filters, subscriptions } ⊃ log streams (one per source)

Retention, access control, metric filters, and subscription filters are all log-group-level settings. Retention defaults to never expire. Metric filters turn log patterns into metrics; Logs Insights queries the raw events.

aws logs put-retention-policy --log-group-name /aws/ecs/api --retention-in-days 30

alarm(metric, statistic, period, evaluation-periods, threshold, treat-missing-data) → SNS | Auto Scaling policy

States are OK, ALARM, INSUFFICIENT_DATA; actions fire on sustained state changes only. The period must be at least the metric resolution. Missing-data behaviour (notBreaching / breaching / ignore / missing) is a deliberate design choice.

aws cloudwatch put-metric-alarm --evaluation-periods 3 --datapoints-to-alarm 2 --treat-missing-data notBreaching ...

metric → how much · log → what happened · trace → where the time went · alarm → what wakes you

Metrics aggregate and cost by cardinality; logs are per-event and cost by volume; traces follow one request across services; alarms are rules over metrics that take actions.

ERROR log lines → metric filter → ApiErrors metric → alarm → SNS

latency(pNN) · error rate(ratio) · saturation(% of limit) · throughput · queue depth · cache hit rate · business KPI

Alarm on user-visible signals plus the saturation signal that actually binds the service, and keep one business measure so a system receiving no traffic cannot look healthy.

errorRate = 100 * Sum(HTTPCode_Target_5XX_Count) / Sum(RequestCount)

trace = segments (per service) + subsegments (per downstream call), grouped by Root trace ID

The first X-Ray-integrated service adds `X-Amzn-Trace-Id: Root=…;Parent=…;Sampled=…`; downstream services propagate it. Uninstrumented downstreams appear as inferred segments. Trace and service graph data are retained 30 days.

X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1

sampling (1/s + 5%) · annotation = indexed (≤50/trace) · metadata = stored, not indexed · group = saved filter + metrics

Sampling makes traces a survey, not a log. Annotate low-cardinality dimensions you will filter by; put everything else in metadata. Errors are 4xx, faults 5xx, throttles 429.

annotation.tenant_id = "acme-4471" AND fault

X-Ray SDK (AWS-native segments + X-Amzn-Trace-Id) · OpenTelemetry → OTLP endpoint → CloudWatch Application Signals

Both produce traces you read in the same console. X-Ray costs least to adopt on an all-AWS stack; OpenTelemetry costs more setup and buys portable instrumentation, OTLP metrics with up to 150 labels, and PromQL queries and alarms.

OTEL_EXPORTER_OTLP_ENDPOINT=<AWS OTLP endpoint> OTEL_TRACES_EXPORTER=otlp

total = own code + db subsegments + downstream subsegments + (round trip − downstream duration)

The upstream subsegment measures the round trip; the downstream segment measures the work. Their difference is network, queueing, and connection acquisition. Time not covered by any subsegment is your own code.

4,200 ms total = 120 ms own code + 3,900 ms SQL + 180 ms auth (of which 60 ms is not work)

app → stdout / agent → log group (per service) → query across the fleet

Never treat an instance or container filesystem as the record. Containers use the task log driver (awslogs); EC2 uses the CloudWatch agent; Lambda writes to CloudWatch Logs by default.

ECS task definition: "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/aws/ecs/api" } }

app logs · ALB/CloudFront access logs (S3) · CloudTrail (S3) · VPC Flow Logs (CW/S3/Firehose) · service logs

Each source answers a different question and has its own destination and retention. Flow logs record flow metadata, never payload, and are collected outside the traffic path. CloudWatch Logs retention defaults to never expire.

aws logs put-retention-policy --log-group-name /aws/vpc/flowlogs --retention-in-days 14

log group = classified data store · permissions at group level · redact at source · KMS for encryption at rest

Access to streams is controlled at the log group. Keep sensitive values out rather than masking them at read time, and give workload roles append-only permissions on their own group.

Task role: logs:CreateLogStream, logs:PutLogEvents on arn:aws:logs:…:log-group:/aws/ecs/api:*

definition in version control → reviewed diff → applied → reality matches definition (else: drift)

Reproducible means rebuildable elsewhere from the file; reviewable means the change is read before it happens. Both require that the code is the only path to production.

aws cloudformation create-change-set --stack-name api-prod --template-body file://api.yaml --change-set-name review-me

template { Parameters, Mappings, Conditions, Resources, Outputs } → stack · change set before update

A stack manages its resources as one unit and deletes them with itself unless DeletionPolicy: Retain is set. Change sets show proposed changes, including replacements, but do not predict whether the update will succeed.

aws cloudformation create-change-set --stack-name api-prod --template-body file://api.yaml --change-set-name review

app ⊃ stacks ⊃ constructs (L1 | L2 | L3) —cdk synth→ CloudFormation template —cdk deploy→ stack

CDK is a generator over CloudFormation. L2 constructs are the intended level; assets go to S3/ECR; a stack's env is its account and Region, and is required for account-specific lookups.

cdk synth · cdk diff · cdk deploy ApiProd

provider + resources + modules → state (remote, locked, encrypted) · plan → apply

State maps configuration to real objects and is what Terraform reads to decide changes. Store it in a remote backend with locking and secure access; never in version control. Plan is the change set; apply executes it.

terraform { backend "s3" { bucket = "acme-tfstate", key = "prod/api/terraform.tfstate", encrypt = true } }

one tool deep (failure handling) + the others as vocabulary · one owner per resource, always

CloudFormation for an all-AWS estate with nothing extra to run; CDK for typed, reusable constructs written by application developers; Terraform for multi-platform estates. Never split ownership of a resource across tools.

Deep = recovering UPDATE_ROLLBACK_FAILED, importing a resource, moving one between stacks

pipeline ⊃ stages ⊃ actions (source | build | test | deploy | approval | invoke) · artifacts flow between them

A stage is locked while processing one execution; the default SUPERSEDED mode lets a newer execution replace a waiting one. Stages can be retried and rolled back to a previous successful execution, automatically on failure.

Build once → sha256:9f2c… → deploy the same digest to staging and production

CodeBuild (run) · CodePipeline (orchestrate) · CodeDeploy (shift traffic, roll back) · OIDC for external CI

All Lambda and ECS CodeDeploy deployments are blue/green; only EC2/on-premises supports in-place. External CI systems should assume an IAM role via OIDC rather than hold long-lived access keys.

role-to-assume: arn:aws:iam::111122223333:role/github-deploy

rolling | blue/green | canary | linear | all-at-once | immutable | feature-flagged

All Lambda and ECS CodeDeploy deployments are blue/green, with canary, linear, or all-at-once traffic shifting. Rolling and canary require backward compatibility; blue/green does not. A canary without an alarm is only a slower deploy.

CodeDeployDefault.ECSCanary10Percent5Minutes — 10% now, the rest after 5 minutes

expand → dual-write + backfill → switch reads → contract · migrations as a pipeline step, not on startup

Rolling and canary deployments run two versions against one schema, so additive changes are safe and renames/drops are not. Code rolls back; data does not — plan both before the release.

R1 ADD COLUMN total_cents · R2 dual-write + backfill · R3 read new · R4 DROP COLUMN total

IAM principal → GetAuthorizationToken (12h) → docker login → repository (tag: mutable · digest: not)

ECR is a private registry with IAM identity and repository policies. Enable tag immutability so a tag cannot be repushed, and reference images by digest in anything that deploys.

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin <acct>.dkr.ecr.eu-west-1.amazonaws.com

lifecycle policy { rules: select by tagStatus / prefix / age or count → expire } · promote by digest

Expire untagged images first — they are the bulk. Keep release tags long enough to roll back to any version you might still want. Promotion deploys the same digest to the next environment.

{"tagStatus":"untagged","countType":"sinceImagePushed","countUnit":"days","countNumber":7}

provenance (cache + signing) · minimal base · scan on push + rescan · USER non-root · multi-stage for secrets

Scan-on-push is a point-in-time statement, so rebuild regularly and keep the base small. Containers run as root unless the image sets USER. Build secrets remain in layers unless a build-secret mount or a discarded build stage keeps them out.

RUN useradd --uid 10001 app && … ; USER app

task definition (revision) → task → service (desired count) → target group (health) → listener rule → ALB

ECS keeps tasks running; the target group decides which receive traffic. ECS resolves the image tag to a digest and reuses it for every task, so all tasks in a service run identical images.

aws ecs describe-tasks --cluster prod --tasks <arn> --query "tasks[].stoppedReason"

minimumHealthyPercent / maximumPercent · circuit breaker (tasks cannot start) + alarms (tasks misbehave) → rollback

Both failure detectors can be used together, and the deployment fails as soon as either one's criteria are met. Secrets are resolved at task start by the execution role; the task role is for the application's own API calls.

deploymentCircuitBreaker={enable=true,rollback=true}, minimumHealthyPercent=100, maximumPercent=200

ALB (public) → tasks (private) → RDS + ElastiCache (private) · SGs reference SGs · secrets by ARN · migrate as a task

The standard three-tier ECS shape. Egress from private tasks goes through NAT or VPC endpoints. Migrations run as a one-off task before the service update, and the deployment enables both the circuit breaker and alarm-based failure detection.

aws ec2 authorize-security-group-ingress --group-id sg-db --port 5432 --source-group sg-task

sync: APIGW → Lambda → DynamoDB · async: S3 / EventBridge / SQS → Lambda → DLQ · queue + reserved concurrency for bounded downstreams

The event source determines the retry and failure semantics, so choosing it is choosing the reliability model. A queue plus reserved concurrency turns a scaling spike into added latency rather than a downstream outage.

API Gateway → Lambda (enqueue) → SQS → Lambda (reserved concurrency 10) → RDS

900 s · 128–10,240 MB (1 vCPU @ 1,769 MB) · 1,000 concurrency/Region · 6 MB sync / 1 MB async · at-least-once

Limits shape the architecture rather than being tuned later. API Gateway's 10,000 rps default exceeds Lambda's 1,000 concurrency, so the front door can admit more than the compute serves. Idempotency is required, not optional.

aws lambda put-function-concurrency --function-name order-worker --reserved-concurrent-executions 10

fits: short + bursty + stateless · does not fit: >15 min, saturated, connection-holding, slow-init runtimes

Lambda is designed for short-lived tasks that do not rely on state between invocations. Per-millisecond billing is cheap when idle and expensive when saturated; VPC-attached functions also consume ENIs, which are quota-limited per VPC.

Can it finish in <15 min? Is it bursty? Is it stateless? Three yeses → serverless.

peering (2, non-transitive) · TGW (many + on-prem, transitive) · gateway EP (S3/DDB, free) · PrivateLink · DX / VPN · NAT

Each transit gateway attachment associates with exactly one route table; VPC and peering attachments need static routes while VPN and Direct Connect propagate via BGP. Gateway endpoints are free route-table entries; interface endpoints are billed ENIs.

Spoke: 0.0.0.0/0 → TGW attachment; TGW route table: 0.0.0.0/0 → egress VPC with the shared NAT

hub (TGW) + spokes · one route per spoke · segmentation via route-table associations · one non-overlapping address plan

Transit gateway route tables and associations segment which spokes can reach which. Overlapping CIDRs cannot be routed between at all, and a VPC's primary CIDR cannot be changed — so allocate centrally, generously, and before the first VPC exists.

10.20.0.0/16 prod-app · 10.21.0.0/16 prod-data · 10.60.0.0/16 dev · 172.16.0.0/12 reserved for on-premises

public: 0.0.0.0/0 → IGW · private: 0.0.0.0/0 → NAT (same AZ) + gateway EPs · data: local only

A subnet is public only because of its route to an internet gateway. Route each private subnet to its own AZ's NAT so an AZ failure stays contained, and give the data tier no default route so it cannot initiate outbound traffic.

aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-0a1b2c3d

1 DNS · 2 route · 3 security groups (both sides) · 4 NACLs (both directions + ephemeral) · 5 listener

Check in order; the first failure is the cause. Timeout means the packet did not arrive or the reply did not come back; connection refused means it arrived and nothing was listening. Flow logs separate REJECT from no-record-at-all.

nc -vz db.internal 5432 → "refused" = listener problem · hang = route / SG / NACL

redundancy + health checks (no shared dependency) + automated replacement + tested failure modes

Availability is bounded by the least redundant component in the path. Health checks must not depend on a shared downstream, or one failure removes every replica at once. Untested failover is a hypothesis.

For every component: how many? which AZ? what happens to the others when that AZ fails?

timeout · bounded retries + jitter · circuit breaker · bulkhead pools · degrade · backpressure · idempotency

A slow dependency is worse than a dead one, because it consumes the caller's capacity for as long as it hangs. Bound the wait, bound the retries, spread them with jitter, isolate pools per downstream, and prefer a reduced answer to an error.

timeout=0.8s, 3 attempts, sleep = uniform(0, 0.1 * 2**attempt), fallback to cached data

quota = hard | soft, per account per Region · enumerate → compare to peak → raise early → alarm on utilization

A quota becomes a reliability constraint at exactly the traffic level you were scaling to serve. Mismatches between services (API Gateway 10,000 rps vs Lambda 1,000 concurrency) are documented by AWS and are a common source of them.

aws service-quotas list-service-quotas --service-code lambda

loosely coupled · throttle · bounded retries · fail fast · timeouts everywhere · stateless services

AWS defines reliability as performing the intended function correctly and consistently when expected, including operating and testing the workload through its lifecycle. Trade-offs between pillars are business decisions; security and operational excellence are generally not traded off.

Six questions: optional dependencies? behaviour above capacity? retry limit? timeout? state? last failover test?

backup/restore < pilot light < warm standby < hot standby < multi-site active/active

Pilot light cannot serve without action first; warm standby serves immediately at reduced capacity. Failover should use only data plane operations, because control planes have lower availability design goals.

Aurora global database (write global) · DynamoDB global tables (write local) · S3 bi-directional replication (write partitioned)

RPO = data loss window (backup/replication frequency) · RTO = downtime window (detect + decide + restore + cut over)

Continuous replication drives RPO toward zero without improving RTO. Both are per-workload business decisions, and both are only useful if checked against the mechanism that would have to deliver them.

payments RPO 5 s / RTO 5 min · internal admin RPO 24 h / RTO 8 h

automated restore tests → integrity checks → staged drills → full DR drill → production failover

AWS recommends regular testing and periodic restores, partly because restore is a control plane operation that may be degraded during the disaster. Automate the failover steps; keep the failover decision human.

Restore into an isolated account, run integrity checks, emit RestoreSucceeded and RestoreDurationSeconds, alarm on both

multi-AZ: synchronous, automatic, baseline · multi-Region: asynchronous, deliberate, per-workload decision

AWS notes that for the loss of a single data centre in a well-architected workload, backup and restore may suffice; pilot light, warm standby, and active/active are for Region-level disasters or regulatory requirements.

Sequence: multi-AZ → cross-Region backups → pilot light/warm standby → active/active

add capacity (out/up/auto) · remove work (cache, CDN, batch, async) · remove contention (pool, partition, isolate)

AWS defines performance efficiency as using resources efficiently to meet requirements and maintaining that as demand changes. Pick the lever from the bottleneck category; scaling out a contention bottleneck degrades it further.

CPU 25% + 3.7 s connection wait → pooling and caching, not more instances

breakdown = own code + downstream + database + resource wait · bottleneck = 100% utilized with work queued

Measure before scaling: the same symptom has different fixes depending on whether the constraint is capacity, contention, or a single slow operation. Percentiles and saturation metrics are what distinguish them.

p99 4,200 ms = 120 own + 270 downstream + 40 query + 3,860 pool wait → contention

concurrency ≈ throughput × latency · saturation → queue depth → latency → timeouts · tail amplifies with fan-out

Six terms: latency (one request), throughput (per second), concurrency (in flight), queue depth (waiting), saturation (percent of a limit), tail latency (p99). Averages hide the tail, and the tail is what users experience.

500 rps × 0.2 s p99 = 100 in-flight → size the pool at 100, not at the average

production-equivalent quotas + data + traffic mix · ramp for the knee · hold for the breakage · confirm against production

AWS recommends load testing to find the quotas that will limit expected traffic. The useful output is which resource saturated first, not the peak requests-per-second figure.

500 rps p99 340 ms (pool wait 95 ms) → knee; 600 rps p99 2.9 s → saturated on connections

compute (time × size) · storage (bytes × time) · requests · provisioned (paid idle) · data transfer (per GB) · managed premium

Read a service by its pricing dimensions the way you read it by its quotas. Data transfer and provisioned-but-idle capacity are the two dimensions that most often surprise, and both are architecture decisions rather than settings.

Lambda: 2M × 300 ms × 512 MB (pay while running) vs Fargate: 2 tasks × 720 h (pay all month)

tag → attribute (Cost Explorer / CUR) → clean up + rightsize → lifecycle storage → commit (SP/RI) → Spot

AWS Budgets tracks cost and usage with alerts and optional actions, and is updated up to three times a day with billing delay on top — so it signals rather than caps. Cost allocation tags apply forward only.

Budget: monthly cost, alert at 80% actual and on forecast-to-exceed, action = restrictive IAM policy

NAT + endpoints · cross-AZ traffic · internet egress · database sizing · log ingestion · idle non-production

Architecture decisions choose the pricing dimensions the workload is billed on, and the resulting saving is larger and more permanent than any commitment. Rank opportunities by how much of the spend should not exist.

Add gateway endpoints for S3/DynamoDB — no hourly or per-GB charge, removes that traffic from NAT processing

sustainability = maximize benefit from provisioned resources + minimize total resources required

One of the six Well-Architected pillars, defined in engineering terms. Under shared responsibility the facility is AWS's concern and the workload's utilization is yours. Track efficiency ratios rather than absolute totals.

requests served per provisioned vCPU-hour · GB actively read per GB stored

utilization → right-size → schedule → lifecycle storage → delete unreferenced data → shrink payloads

The same levers as cost work, read as resources provisioned versus resources used. Storage is the one that grows without anyone deciding to, so it needs lifecycle automation rather than periodic review.

aws ec2 describe-volumes --filters Name=status,Values=available — provisioned, powered, attached to nothing

6 pillars, traded per workload · sustainability ≈ cost · tensions with reliability and performance headroom

AWS documents trade-offs between pillars as business decisions, gives development (sustainability and cost over reliability) and mission-critical (reliability over both) as examples, and notes security and operational excellence are generally not traded off.

Warm standby: RTO 10 min instead of 4 h, +$6.4k/month, a duplicate environment always running — accepted, and recorded

operational excellence · security · reliability · performance efficiency · cost optimization · sustainability

A consistent set of best practices expressed as questions for evaluating an architecture. Component / workload / architecture / milestones / technology portfolio are the framework's own vocabulary.

A workload (checkout) = components (API, worker, database) arranged in an architecture, reviewed at milestones

six questions → risks with owner + level of effort (high | medium | low) → reviewed again at each milestone

The framework is a set of questions for evaluating an architecture. Reviews produce prioritised risks rather than a pass/fail, and are run at milestones because the answers change from intentions to observations.

Risk: single NAT gateway · Effort: low · Owner: platform · Status: accepted for staging

tension + decision + what was given up + assumptions + owner + review trigger

AWS frames pillar trade-offs as business decisions driven by context. Recording the assumptions is what makes a decision reviewable, since assumptions expire quietly while the architecture persists.

Reliability vs cost: warm standby, +$6.4k/month, assumes >1 h downtime costs more than $77k/year

runbook (known task, exact commands) · playbook (unknown problem, decision tree) · mitigate → diagnose → postmortem → change

AWS defines operational excellence as running workloads effectively, gaining insight, and continuously improving procedures. A runbook must be executable by someone unfamiliar; postmortem actions need an owner and a date.

Action: "Alarm on target 5xx > 1% for 2 min — owner: payments — due: 12 Sep"

manual → runbook → script → triggered · automate the mechanism, keep the decision · alarm on failure and on absence

A precise runbook is most of the work of a script. Automation removes variance before it removes time, and needs the same review, least-privilege permissions, and monitoring as application code.

Scheduled restore test → emit RestoreSucceeded → alarm on failure and on no run in 26 hours

documented + measurable + testable · every change: how deployed, how observed, how reversed

Operational readiness asks whether something can be run in production, before it is. Readiness and correctness reviews find disjoint problems, and rollback paths decay whenever the pipeline changes.

Go-live: test page received? alarm per user-visible failure? rollback performed in staging? restore executed?

account = the isolation boundary · crossing it needs a trust policy, a resource policy, or a share — never a default

SCPs do not affect the management account, so workloads there sit outside every organization guardrail. Centralize human identity with IAM Identity Center while keeping accounts as the blast-radius boundary.

Trust policy: Principal arn:aws:iam::111122223333:role/deploy-pipeline, Action sts:AssumeRole, with an ExternalId condition

SCP guardrails + Identity Center + log archive account + read-only security account + alarmed break-glass

AWS recommends testing SCPs on an OU rather than the root, and warns that removing FullAWSAccess without a replacement fails every action from member accounts. Service last accessed data is how guardrails get tightened with evidence.

Deny SCP on cloudtrail:StopLogging, guardduty:DeleteDetector, config:DeleteConfigurationRecorder

effective = SCP ∩ permissions boundary ∩ identity/resource policy · explicit Deny always wins · SCP grants nothing

AWS states that no permissions are granted by an SCP; it defines a guardrail on the maximum available permissions. SCPs do not apply to the management account or to service-linked roles.

SCP allows s3:*, role has no policy → no access. SCP allows s3:GetObject, role allows s3:* → GetObject only.

encryption (read) · backups (corruption) · replication (location) · versioning + immutability (deletion) · testing (all)

AWS Backup centralizes scheduling across services and copies across Regions and accounts, which it names as protection against insider threats and account compromise. Replication propagates corruption; only point-in-time backups do not.

Production account → AWS Backup copy → dedicated backup account with immutable retention, unreachable from production roles

sensitivity (leak) and availability (loss) are independent dials · derived data inherits both

Sensitivity drives keys, access scope, and logging; availability drives replication, backup frequency, and per-data RPO/RTO. Tag the classification so policy can act on it, and apply it to copies as well as sources.

DataClassification = public | internal | confidential | restricted; AvailabilityTier = tier1 | tier2 | tier3

hot → warm (IA) → cold (archive) → expire · plus NoncurrentVersionExpiration whenever versioning is on

Lifecycle rules transition and expire by age or tag without anyone running them. Colder classes carry minimum storage durations and retrieval costs, so tiering short-lived or numerous small objects can cost more than it saves.

Transitions: 30 d → STANDARD_IA, 90 d → GLACIER; Expiration 1095 d; NoncurrentVersionExpiration 30 d

Route 53 → CloudFront/WAF → ALB (public) → ECS/EC2 (private) → RDS + ElastiCache + S3

The default shape for a server-rendered or API-backed web application. Only the ALB is public; security groups chain by reference; the app tier is stateless so any target can serve any request.

sg-app inbound 8000 from sg-alb · sg-rds inbound 5432 from sg-app

Route 53 → API Gateway → Lambda → DynamoDB/S3 · EventBridge/SQS → worker Lambda

Synchronous path kept short and retry-safe; slow work published as an event. API Gateway defaults to 10,000 rps, Lambda to 1,000 concurrent executions, 15-minute timeout, 6 MB sync payload.

PutItem with ConditionExpression: attribute_not_exists(order_id) → a retried POST cannot create two orders

Producers → EventBridge/SNS → one SQS queue per consumer → workers → data stores

Fan-out by rule or subscription, buffer per consumer, dead-letter queue per consumer. Producers publish past-tense facts and never learn who consumes them; consumers are idempotent and eventually consistent.

OrderPlaced → {billing-work, search-index-work, notify-work} — adding a fourth consumer redeploys nothing upstream

S3 | Kinesis → raw zone (immutable) → transform → curated zone (partitioned Parquet) → analytics

Raw is the only copy that cannot be rebuilt, so nothing edits it. The transform writes whole partitions, which makes reruns and backfills safe. Curated is partitioned by the queried column and read by everything downstream.

raw/orders/ingest_date=… (rerun by this) · curated/orders/order_date=… (query by this)

API (presigned URL) → browser PUT → S3 → event → SQS → worker → processed object → notify

Bytes never pass through your compute. Presigned URLs carry the signer's credentials and expire with them — up to 7 days for IAM user credentials, sooner for role sessions. Events and queue delivery are at least once.

uploads/{tenant}/{job_id}/{name} → conditional claim on job_id → processed/{tenant}/{job_id}/{name}

DATABASES→RDS · CACHES→ElastiCache · media→S3 · broker→SQS · secrets→Secrets Manager · logs→CloudWatch

One container image, two ECS services (web behind an ALB, worker with no ports). Credentials come from the task role; secrets are injected by ARN; RDS Proxy pools connections when the compute model opens many of them.

orders:1.4.2 + command gunicorn → web · orders:1.4.2 + command celery worker → worker

static → build-time, public, immutable · media → runtime, private, presigned · migrate → one-off task, expand then contract

Static and media are stored and served differently. Uploads bypass the application via presigned PUT. Migrations run as a gated task, split expand/backfill/contract because both versions serve during a rolling deploy. Health checks touch no dependency.

/healthz → ALB target group (no dependencies) · /readyz → alarms only (checks DB and cache)

web → requests per target / p95 latency · worker → backlog per worker (queue length ÷ workers in service)

Web and worker load peak at different times, so they need separate services and separate policies. Target backlog per worker = acceptable latency ÷ average processing time; AWS's example is 10 s ÷ 0.1 s = 100 messages.

1,500 messages ÷ 10 workers = 150 backlog per worker, against a target of 100 → scale out

files → S3 (private) + CloudFront + OAC · server → ECS/Lambda behind CloudFront · SPA → 403/404 ⇒ /index.html 200

Static output needs no compute; CloudFront reaches a private bucket via origin access control, which AWS recommends over OAI and which requires Object Ownership "Bucket owner enforced". A website-endpoint bucket must be a custom origin and cannot use OAC.

/_next/static/* → S3 origin, cached a year · /* → ALB origin, not cached

alias record · ACM in us-east-1 · hashed assets immutable · invalidate /index.html only · /api/* same origin

A CloudFront certificate must be requested or imported in us-east-1, and the alternate domain name must be covered exactly or by a same-level wildcard in the SAN field. Serving the API as a behavior on the same distribution removes CORS and cross-site cookies entirely.

/assets/app.a91f3c.js → max-age=31536000, immutable · /index.html → no-cache · /api/* → no-store

Browser → Route 53 → CloudFront → (S3 assets | ALB → backend → RDS)

One domain, two paths. Assets are cached at the edge from a private S3 origin; API calls are forwarded uncached with the Authorization header. Sessions ride an HttpOnly first-party cookie when the API is same-origin.

Set-Cookie: session=…; Secure; HttpOnly; SameSite=Lax; Path=/

steady state → hypothesis → experiment (scoped, with stop conditions) → verify → improve → automate

AWS: chaos engineering is experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. Baseline on user-visible output, not internal metrics, and never run an experiment you already expect to fail.

"If one of four tasks is stopped, 5xx stays under 0.05%" — a hypothesis that can be wrong, and was

instance · AZ · container · worker · queue backlog · cache · DB failover · 3rd-party timeout · DNS · credentials · rollback

The recurring failure list, prioritised by frequency × damage from past incidents. FIS ships preconfigured faults for termination, forced failover, CPU and memory stress, throttling, latency and packet loss across EC2, ECS, EKS and RDS.

Database failover drill: the database recovers in seconds; the connection pool holding dead connections is what extends the outage

FIS experiment template = actions + targets + stop conditions (≤ 5 CloudWatch alarms)

FIS runs real faults on real resources. Targets can be selected by tag or state, which is how blast radius is bounded; a triggered stop-condition alarm stops the experiment, and a post action can return targets to their prior state.

resourceTags {Service: orders, Chaos: eligible} · selectionMode COUNT(1) · duration PT5M · alarm 5xx > 1%

symptom → recent change → metrics → logs → traces → dependencies → network → IAM → quotas → cost

A search order chosen by how much each step eliminates. Metrics locate, logs describe, traces attribute latency to a hop. Stop at the first step that fully explains the symptom.

"POST /orders, 502, 12%, since 14:05 — GETs unaffected" — the last clause eliminates DNS, TLS, CDN and the load balancer

5xx · latency · Lambda throttles · ECS restarts · CPU · RDS connections · SQS backlog · NAT ports · DNS · IAM · SG

Each investigation is a metric, an error and a first check. NAT allows 55,000 simultaneous connections per destination per IP, times out idle connections after 350 seconds, and exposes ErrorPortAllocation and IdleTimeoutCount.

HTTPCode_ELB_5XX (no healthy target) vs HTTPCode_Target_5XX (the app returned it) — one look, two different incidents

correlation_id (yours, every line, crosses queues) · X-Amzn-Trace-Id (X-Ray, sampled) · AWS request id (one API call)

The tracing header carries Root, optional Parent and the sampling decision, added by the first X-Ray-integrated service. X-Ray samples the first request each second plus 5% of the rest, so logs — not traces — are what give complete per-request coverage.

filter correlation_id = "c-9f13ab77" | sort @timestamp asc — across every log group, sync and async

default → (adjustable?) → applied quota · usage ÷ quota = utilization · global quotas raised from us-east-1

A service quota is the maximum resources or operations for an account or a Region. The console shows an "Adjustable" column; increases are reviewed and may be approved, denied or partially approved. New accounts start with reduced Lambda concurrency and memory quotas.

aws service-quotas get-service-quota --service-code lambda --quota-code L-B99A9384

Service Quotas utilization alarm @ ~70% (warning) + service throttling alarm @ >0 (backstop)

Service Quotas can create a CloudWatch alarm on a quota to notify you as you approach its value, and shows utilization once a quota is in use. Increases are per quota per Region, are reviewed, and may be approved, denied or partially approved.

utilization = usage ÷ quota — 150 of 200 is 75%, and stays meaningful when the quota becomes 500

Lambda 900 s / 6 MB · NAT 55,000 per destination per IP · API GW 10,000 rps vs Lambda 1,000 concurrent · SQS 256 KB

Adjustable limits are a request; fixed limits are design inputs. The Lambda timeout and payload ceilings cannot be raised, subnet CIDRs cannot be resized, and total database connections are derived from your own pool sizes.

40 ECS tasks × 20-connection pool = 800 connections — 80% of a db.r6g.large before the worker tier is counted

Application · Environment · Owner · CostCenter · DataClassification · ManagedBy

Keys and values are case sensitive, so one spelling per key is non-negotiable. Tag policies in Organizations standardise case treatment and can enforce compliance on specified resource types. Never store personal or confidential data in a tag.

Environment=prod (not Production, not ENV=prod) · Owner=team-payments (not a person)

cost allocation · incident ownership · automation selection · IAM conditions · compliance scope

User-defined and AWS-generated cost allocation tags are activated separately in the Billing console, only by the management or a standalone account, and can take up to 24 hours to appear. The cost allocation report includes untagged resources, so coverage gaps stay visible.

Condition: StringEquals aws:ResourceTag/Environment = dev — a policy that stays correct as resources come and go

document → default in IaC → report compliance → enforce narrowly → enforce widely

Tag policies need an organization with all features enabled. AWS recommends understanding a policy's effect on one account before widening it, and especially before enforcing — enforcement prevents noncompliant tagging requests on specified resource types from completing.

<app>-<env>-<component> → orders-prod-rds-writer · facts live in tags, not in the name

desired state (template) ≠ actual state (account) ⇒ drift · resolve by updating, redeploying, importing, or recording

A resource has drifted if any actual property value differs from the expected value, including deletion; a stack has drifted if any resource has. Only explicitly set properties are compared, stack-level tags are included, and nested stacks need their own detection run.

A property left out of the template is never compared — set it even when the value equals the default

DRIFTED · IN_SYNC · NOT_CHECKED · (resource) MODIFIED, DELETED · (property) ADD, REMOVE, NOT_EQUAL

Detection needs read permission per resource type plus DetectStackDrift, DetectStackResourceDrift and BatchDescribeTypeConfigurations. Nested stacks are not covered by a parent run, KMSKeyId is never compared, and unset properties are excluded.

IN_SYNC 31 · DRIFTED 0 · NOT_CHECKED 19 — the third number is the coverage gap

replace, do not modify · definition in version control · console read-only by default, writes via an assumed role

Immutable infrastructure means a change is a new version plus a replacement, so every running resource is reproducible. Changes made outside the tool complicate later stack updates and deletions, which is why every manual change ends in promote-or-revert.

Failing instance → terminate and let the ASG replace it, not SSH and patch it

separate role · alerts on use · bounded scope · closes with drift detection and promote-or-revert

A sanctioned manual path for when the normal one cannot work, written before it is needed. It must function during an identity provider or pipeline outage, and it is tested on a schedule like any other recovery procedure.

Assume BreakGlassProd (MFA, alerts #sec-alerts) → one change → record at the time → drift detect → promote or revert

client-supplied key + atomic record-and-mutate + stored response on retry + retention beyond resource lifetime

AWS: an idempotent operation is one where a request can be retransmitted or retried with no additional side effects, and the service returns semantically equivalent responses. Recording the token and the mutation must together be atomic, consistent, isolated and durable.

PUT is naturally idempotent; POST /orders needs an Idempotency-Key the client generates once and reuses on every retry

idempotency key · conditional write · unique constraint · FIFO dedup (5 min) · deterministic key · durable state

DynamoDB attribute_not_exists rejects a second create with "The conditional request failed". SQS FIFO does not introduce duplicates when SendMessage is retried within the 5-minute deduplication interval, using either a content hash of the body or an explicit deduplication ID.

put-item --condition-expression "attribute_not_exists(Id)" — the failure is the success case for a retry

at-least-once delivery (theirs) + idempotent effect (yours) = exactly-once outcome

Exactly-once delivery is unavailable because a sender cannot distinguish a lost request from a lost response. SQS FIFO deduplicates SendMessage retries within a 5-minute interval — producer-side and time-bounded — and standard queues deduplicate nothing.

Visibility timeout expires mid-call → the message is redelivered → two charges, with no component having failed

requirements → scale → API → data model → services → failure and operations

A derivation, not a recollection. Numbers before boxes: requests per second, object size, growth and read-to-write ratio decide more than any preference. State the consistency allowance explicitly, because it authorises and bounds every cache in the design.

100M links, 100:1 read:write, 20 GB → a read-latency problem, which rules out most architectures before any are proposed

shortener→caching · notifications→fan-out · payments→idempotency · SaaS→isolation · ingestion→backpressure

Each recurring system is a known architecture plus one hard part. The test that names it: what would still be difficult with ten users? Correctness problems are hard at any scale; volume problems are not problems until the volume exists.

Image pipeline and payment workflow draw identically and share none of the difficulty

requirements · scale · API · data model · network · IAM · compute · database · cache · queue · failure · observability · cost · security · RPO/RTO · ownership

The Well-Architected pillars are the standard review lens over these dimensions. The four most often missing — failure modes, cost drivers, RPO/RTO and ownership — are the ones with no visible symptom before production.

Failure modes for one dependency: timeout, retry bound, breaker, degraded behaviour, alarm, owner — not "it retries"

privilege · encryption · secrets · network · logging · vulnerabilities

The six security questions in a review, each closed by evidence rather than an assurance. Review the live account configuration, because the diagram does not drift and the account does.

aws ec2 describe-security-groups --filters Name=ip-permission.cidr,Values=0.0.0.0/0

multi-AZ · health checks · backups · failover · retries · idempotency · degradation

The reliability pass, written as failures rather than features. Each failure needs a control and a date the control was last exercised; a control with no date is a plan.

Zone loss → 2 AZs with capacity for 1 → drilled 2026-08-14, recovered in 41 s

compute · cache · pool · database · CDN · async

The six performance levers, applied only after a latency breakdown says which segment dominates. Async is the one that removes time rather than shortening it.

p95 900 ms → 310 ms is pool wait → resize the pool, not the cache → p95 610 ms

rightsizing · autoscaling · lifecycle · data transfer · NAT and logs · budgets

The cost pass, read by driver rather than by total. Capacity spend is shrunk; accumulated architecture spend — NAT processing, log ingestion, wrong storage class — is removed.

aws ce get-cost-and-usage --group-by Type=DIMENSION,Key=USAGE_TYPE

dashboards · alarms · runbooks · deployment safety · incident roles · auditability

The operations pass, governed by one question: could someone who did not build this recover it at 3 a.m.? Each artefact removes one place the incident would otherwise stall.

Hand the runbook to another team and have them work last month's incident from it

utilisation · demand alignment · lifecycle · efficient work

The sustainability pass finds provisioned capacity that is not doing work, and work that is larger than the result requires. It overlaps the cost pass and adds the efficiency of the computation itself.

A nightly full reprocess of 40M rows to update the 12k that changed

Region ⊃ AZ · user vs role · identity vs resource policy · deny > allow > implicit deny

The four opening questions. Each has a one-line definition and a consequence, and the consequence is what is being tested — especially what multi-AZ fails to contain.

AccessDenied on an encrypted object with s3:GetObject allowed → check kms:Decrypt on the key policy

route table → public · stateful vs stateless · NAT out vs endpoint inside · L7 / L4 / API front door

The four networking questions answered by property rather than by service name. The route table is what makes a subnet public; statelessness is the whole NACL difference; an S3 gateway endpoint has no data processing charge.

Reply never arrives with the security group open → NACL missing inbound ALLOW TCP 1024-65535

concurrency = rps × duration · default 1,000/Region · reserved caps · provisioned pre-warms

The compute questions, answered by elimination on execution model. Lambda concurrency counts executions in flight, so a slower dependency raises it without any change in traffic.

500 rps × 0.2 s = 100 concurrent; the same 500 rps × 3 s = 1,500 and throttles

standby (unreadable, automatic) vs replica (readable, lagging) · storage layer · patterns up front · may it vanish

Four data-store pairs, each on its own axis. Naming the axis converts "which is better" into a statement about what the workload needs.

Read-after-write against a replica → intermittent "not found" that never reproduces

SQS pull, one consumer · SNS push, all subscribers · EventBridge content routing + replay · FIFO 300 rps/action

The messaging questions answered on coupling rather than throughput. SNS into SQS is the production default; idempotency is required whichever queue type you pick.

key = sha256(order_id + event_type), conditional write before the side effect, TTL well past any redelivery window

BPA first · what is dynamic · migrations as their own task · expand/contract · RPO first · edge→inward · group by usage type

The seven open scenario questions, each with a fixed opening move. The move is what makes the answer structured; naming the constraint is what makes it defensible.

"What RPO and RTO has the business agreed?" — asked before drawing any DR architecture

tier 1 = design + debug + defend · tier 2 = build with docs open · tier 3 = recognise and name the cost

The seventeen tier-1 items are foundations, not products, and every incident is debugged through them. Mark yourself by verb, group gaps by root cause, and close the one that blocks you most often.

"Tier 2 on DynamoDB, missing hot partitions" is a study plan; "learn DynamoDB better" is not

Route 53 → ALB → ECS (web + workers) → RDS + Redis + S3, secrets by ARN, migrations as a gated task

The standard containerised backend, built in seven demonstrable milestones. Execution role pulls and reads secrets; task role is what the code uses.

aws ecs run-task … migrate → wait → check exit code 0 → aws ecs update-service

API Gateway → Lambda → DynamoDB, EventBridge/SQS for the rest, DLQ on every async path

The serverless API built around its joins: what fits in 29 seconds, what becomes an event, and what happens when a consumer runs twice.

Return 202 with an id, publish OrderReceived, let email, search and billing consume it separately

presigned PUT → S3 event → SQS → worker → derived key → notify · DLQ + alarm

The file pipeline, where the design work is entirely in the two failure paths: a file that can never be processed, and a file that takes longer than the visibility timeout.

p99 90 s → visibility timeout 300 s → function timeout 240 s → maxReceiveCount 3 → DLQ

context · authz · RLS · namespaced cache keys · S3 prefix · per-tenant limits · tenant as a dimension

Multi-tenancy is the same architecture as a single-tenant API with tenant identity threaded through seven places. Every one is cheap up front and a migration afterwards.

SET LOCAL app.tenant_id = '3f0c…' with a USING policy, so a forgotten filter returns zero rows

2 AZs with 1-AZ capacity · private data · endpoints off NAT · logs elsewhere · IaC only · 3 drills

The full platform, defined in code and proved by three drills: zone loss, timed restore, and a rebuild into an empty account. The third finds the most and is skipped the most.

Restore drill: 22 min for the database, 40 min to reconfigure — the real RTO was 62 min

foundations → network → compute → storage → data → decoupling → operating → synthesis

The thirty-five steps read as eight groups. Groups 1 and 2 are hard prerequisites for everything else; each group should end in something you built rather than something you read.

Group 3 is done when you terminated an instance and watched the fleet heal

four numbers + an agreed RPO/RTO → accounts → network → path → state → identity → observability → delivery → recovery

The exit test for the topic: derive a full platform design from an unquantified paragraph, then perturb it. A design that produces the same diagram for every requirement was recalled rather than derived.

"RPO 5 min, RTO 15" changes backup-and-restore into warm standby and nothing else in the design

requirements → accounts → network → IAM → compute → LB → data → cache → queues → security → observability → IaC → deploy → scale → cost → recovery

The sixteen-step order, grouped into three widening bands. Each step is decided using the answers above it, and the two orderings most often reversed are cache-before-store and IaC-before-observability.

A data-residency line added at step 1 moves fourteen of the sixteen steps