Filter concepts by levelShowing all levels.

AWS · Section 70

Target Outcome

Level
advanced
Read
30 min
Concepts
2

The exit test for the topic is a paragraph and a method. The paragraph asks for a high-concurrency platform with a relational database, a cache, file uploads, asynchronous workers, authentication, monitoring, autoscaling, continuous delivery and disaster recovery — and it contains no numbers at all, which is the first thing to notice. Turning "high-concurrency" into requests per second, "file uploads" into an object size, and "disaster recovery" into an agreed RPO and RTO is what converts a paragraph into constraints, and several architectures are defensible until those constraints exist. The method is a sixteen-step order that reads as three widening bands: where things live and who may touch them, what serves the traffic and holds the state, and how the result is watched, shipped, paid for and recovered. The order is load-bearing rather than tidy — data stores are chosen before caching because a cache is an optimisation over a store and choosing it first hides which store was needed, and observability is designed before infrastructure as code so the alarms ship with the resources instead of being added by hand afterwards and lost in the next rebuild. The final test of both is perturbation: change one line of the requirement and say precisely which components move and which do not. A design that produces the same diagram whatever the requirement says was recalled; a design where a data-residency sentence moves fourteen of the sixteen steps was derived.

What is true here

  1. The requirement paragraph has no numbers, and asking for them is the first move.
  2. Sixteen steps in three bands: boundaries, then the system, then how it is run.
  3. Data stores before caching; observability before infrastructure as code.
  4. Failure recovery is last and tests every decision made before it.
  5. Perturbation is the test: change one requirement and name what moves.

What you will be able to do

  • Derive a full platform design from an unquantified requirement paragraph
  • Ask for the four numbers and the agreed RPO and RTO before drawing anything
  • Run the sixteen-step order and justify each step from the ones above it
  • Trace a single requirement change through every step it affects
  • Close a design with the trade-off made and the condition that would reverse it
The exit test, and the order that passes it
answered byrunning the orderchange one requirementand watch it move

One paragraph of requirements

unquantified on purpose — the first move is asking for numbers

Sixteen steps, in order

each decided from the answers above it, ending in failure recovery

  • One paragraph of requirements — unquantified on purpose — the first move is asking for numbers
    • leads to Sixteen steps, in order (answered by running the order)
  • Sixteen steps, in order — each decided from the answers above it, ending in failure recovery
    • leads to One paragraph of requirements (change one requirement and watch it move)

Target Outcome

The requirement you should be able to answer cold, and the sixteen-step order that produces the answer — plus the perturbation test that tells a derived design from a remembered one.

The Requirement You Should Be Able to Answer Cold

coreadvanced

This is the exit test for the whole topic: one paragraph of requirements, and a complete design produced from it in an interview or a design session, with every service justified by something earlier in your own reasoning. Not a diagram you remember — a design you derive, where changing one requirement visibly changes the answer.

Think of it as

The requirement paragraph is a list of constraints in disguise. "High-concurrency" is a number you must ask for. "Asynchronous workers" means a second scaling dimension. "Disaster recovery" means an RPO and RTO nobody has stated yet. Read it as a set of questions first, and the design follows from the answers rather than from a template.

What we're doing: Answer the requirement paragraph out loud, in the order it should come.

answer.txttext
Before I draw anything, four numbers and one agreement:
  peak requests per second, read-to-write ratio, upload size,
  growth over 12 months — and the agreed RPO and RTO.
  Say 400 rps, 90% reads, uploads under 5 MB, 3× growth,
  RPO 1 hour, RTO 4 hours.

ACCOUNTS AND NETWORK
  Separate production and non-production accounts. One VPC per
  environment, two Availability Zones, three subnet tiers:
  public for the ALB and NAT, private for the application,
  private for data. Gateway endpoints for S3 so upload traffic
  does not take the NAT path — that is a cost decision I would
  rather make now than find on a bill.

COMPUTE AND TRAFFIC
  Route 53 → CloudFront → ALB → ECS on Fargate. Two services
  from one image: web running gunicorn, workers running Celery.
  Web scales on request count per target; workers on queue
  depth. Six web tasks at peak, and the Auto Scaling maximum is
  sized so one zone can carry the load, not two.

DATA
  RDS PostgreSQL, Multi-AZ, one read replica for reporting. The
  connection arithmetic: 4 gunicorn workers × 6 tasks × a pool
  of 5 is 120 connections, comfortably under the instance limit
  — worth checking, because it is the number that quietly
  breaks. ElastiCache for sessions, rate limiting and one hot
  read. Redis is never the system of record.

UPLOADS AND STATIC
  Presigned PUT straight to S3, so a 5 MB body never passes
  through the ALB or a task. Static assets in a second bucket
  behind CloudFront with origin access control.

IDENTITY AND SECRETS
  An identity provider for user login, sessions in ElastiCache
  so any task can serve any request. Task role for the app,
  execution role for the pull, secrets by ARN — nothing in the
  image.

OBSERVABILITY
  Four alarms worth paging on: 5xx rate, p95 latency, queue
  depth, database connections. Traces with a correlation id
  from the ALB through to the worker. A dashboard that answers
  "what is broken" in two minutes.

DELIVERY
  Everything in Terraform. Pipeline: build, push to ECR, run
  migrations as a gated one-off task, update the service, watch
  the alarm for ten minutes, roll back by redeploying the
  previous task definition.

RECOVERY
  RPO 1 hour and RTO 4 hours means backup and restore across
  Regions, not warm standby — and I would say plainly that this
  is the cheap option the stated numbers allow. Point-in-time
  recovery on the database, S3 replication, the stack
  deployable into the second Region from the same repository,
  and a restore drilled quarterly with the time recorded.

THE TRADE-OFF I MADE
  A single Region, because the RTO permits it. If that number
  becomes fifteen minutes, the design changes to warm standby
  and the cost goes up materially. That is the one line I would
  want in the document.
1
Asking for the numbers before drawing is the single strongest signal in this whole exercise, and it takes fifteen seconds.
6
Accounts and network first, because every later component is placed inside decisions made here.
14
Two services with two scaling signals is the detail that shows the "asynchronous workers" phrase was read as an architecture requirement rather than a feature.
30
Presigned uploads keep large bodies out of the request path entirely, which removes a whole class of timeout and memory problems.

Why this works: The answer is judged on order and justification rather than completeness. Numbers before boxes, then containment (accounts, network), then the request path, then state, then identity, then how it is watched, delivered and recovered — and a closing statement of the trade-off with the condition that would reverse it. Every component named is traceable to something said earlier in the same answer.

Drawing the architecture before asking for a number

Wrong

text
# "Route 53, CloudFront, ALB, ECS, RDS, Redis, S3, SQS,
#  CloudWatch, CodePipeline, and multi-Region DR."
# Ninety seconds, complete, and identical for any requirement.

Better

text
# "What is peak throughput, what is the read-to-write ratio,
#  how large are the uploads, and what RPO and RTO has the
#  business agreed?" Then the same components — but each one
#  chosen, and two of them different.

What you see: A complete and correct-sounding answer that cannot survive a follow-up, because nothing in it was chosen and so nothing in it can be defended.

Why: The paragraph is deliberately unquantified, and noticing that is part of what is being tested. Without the numbers, several architectures are equally defensible and none can be recommended — so an answer given without them is a recollection rather than a design, and the perturbation test exposes it immediately.

From one paragraph to a defensible design

The middle column is what turns the paragraph into a design. Skipping it produces the same diagram for every requirement, which is the failure this exercise is testing for.

  • Three columns: the requirement phrase, the question it forces, and the component that answers it.
  • High-concurrency: ask for requests per second, payload size, growth and read-to-write ratio; answered by an ALB with an ECS service and target tracking.
  • Django and PostgreSQL: ask how many connections per worker and how failover is handled; answered by RDS or Aurora with a standby, a connection pool sized against max_connections, and a proxy if needed.
  • Redis: ask what is cached and what a miss costs; answered by ElastiCache for cache, sessions and rate limiting, never as the system of record.
  • S3 file uploads: ask how large and who may read them; answered by presigned uploads direct to S3, with CloudFront for delivery.
  • Asynchronous workers: ask what must not block a response; answered by SQS with a separate worker service scaling on queue depth.
  • Authentication: ask whose identities and where sessions live; answered by an identity provider plus a shared session store.
  • Production monitoring: ask which signals would page someone; answered by CloudWatch metrics, logs, alarms and traces.
  • Autoscaling: ask which metric actually tracks load; answered by target tracking on request count for web and queue depth for workers.
  • CI/CD: ask how a bad release is reversed; answered by a pipeline with gated migrations and a tested rollback.
  • Disaster recovery: ask what RPO and RTO were agreed; answered by the strategy those two numbers permit.

The ten requirement phrases, the number or decision each hides, and what changes if the answer changes

The ten requirement phrases, the number or decision each hides, and what changes if the answer changes
PhraseWhat it hidesIf the answer is smallIf the answer is large
High-concurrencyRequests per second at peak, and the read-to-write ratioTwo tasks behind an ALB, one database, no cache yetTarget tracking, read replicas, a cache, and a connection proxy
PostgreSQLConnections per worker × workers × tasks, against `max_connections`A pool per task is fineRDS Proxy or a pooler, because the arithmetic exceeds the engine limit
RedisWhat is cached, and what staleness the requirement allowsA single node for sessions and rate limitingA replicated cluster, plus stampede protection on the hot keys
S3 file uploadsObject size, and whether the files are public or per-userPresigned PUT, private bucket, presigned GET to read backMultipart uploads, lifecycle rules, and CloudFront with signed URLs
Asynchronous workersWhat may finish after the response, and the acceptable delayOne queue, one worker serviceSeveral queues by priority, per-tenant fairness, and a DLQ each
AuthenticationWhose identities, and where the session livesSessions in the shared cache, one identity providerFederation, token refresh, and per-tenant authorization rules
Production monitoringWhich signals would actually make someone actFour alarms: errors, latency, saturation, queue depthTracing, per-tenant dimensions, and a two-minute dashboard
AutoscalingThe metric that genuinely tracks load for each tierTarget tracking on CPURequest count per target for web, queue depth for workers
CI/CDHow a bad release is reversed, including a migrationPipeline, gated migration task, redeploy the previous revisionBlue/green or canary with automatic rollback on an alarm
Disaster recoveryThe agreed RPO and RTO, which nobody has statedBackup and restore, tested quarterlyWarm standby or active/active, with the cost stated plainly

Together

text
# Perturbation: the test that a design was derived, not recalled.

Baseline requirement: 400 rps peak, 90% reads, uploads under
5 MB, RPO 1 hour, RTO 4 hours.

  → ALB + ECS (6 tasks, target tracking on request count)
  → RDS PostgreSQL Multi-AZ, one read replica
  → ElastiCache single replicated node group
  → presigned PUT to S3, CloudFront for delivery
  → SQS + 2 worker tasks scaling on queue depth
  → backup and restore across Regions, drilled quarterly

Now change ONE line and say what moves.

  "RPO is 5 minutes, RTO is 15."
  → backup and restore is out. Warm standby: the stack deployed
    in a second Region at zero desired count, Aurora Global
    Database or cross-Region read replica for the data, Route 53
    health-check failover, and roughly a 40-60% cost increase
    which I would state before proposing it.
  → nothing else in the design changes.

  "Peak is 8,000 rps, not 400."
  → the web tier is arithmetic, so it just gets bigger. The
    database is not: 90% reads at that volume needs several
    replicas and a routing decision in the application, and the
    connection count now exceeds max_connections, so RDS Proxy
    enters the design.
  → the cache stops being optional and becomes load-bearing,
    which means stampede protection is now required.

  "Uploads are 4 GB video files."
  → multipart upload, and the worker moves from Lambda-shaped
    work to ECS tasks with real memory and no 15-minute ceiling.
  → S3 lifecycle and storage class become a cost driver worth
    naming.

If you can do this, the design was derived. If every requirement
produces the same diagram, it was recalled.

Remember: Ask for the numbers before drawing: peak rate, read-to-write ratio, payload size, growth, and the agreed RPO and RTO. Then work outward — accounts and network, request path, state, identity, observability, delivery, recovery — and finish with the trade-off you made and the condition that would reverse it. Then perturb it: change one requirement and say what moves.

See also: designing a system on aws · the reasoning progression · project 1 production django api · rpo and rto · mapping a backend to aws

The Sixteen-Step Reasoning Progression

coreadvanced

Sixteen steps, and the order is the point. Each one is decided using the answers from the steps before it, which is what makes the design derivable instead of remembered. Jumping to compute before the network, or to caching before the data model, produces a design whose pieces cannot explain each other.

Think of it as

Think of it as widening from a boundary to a behaviour. The first four decide where things live and who may touch them. The middle six decide what handles the traffic and holds the state. The last six decide how you watch it, ship it, pay for it and recover it. A step is only finished when its answer constrains the next one.

What we're doing: Watch one requirement change ripple through the sixteen steps.

ripple.txttext
Change at step 1: "this platform now serves regulated health
data in two countries, with data residency requirements."

One line in the requirements. Watch where it lands.

STEP 2 · accounts        Per-region production accounts, not one.
                         The isolation is now a compliance control
                         and not a preference.
STEP 3 · network         Separate VPCs per Region, no peering
                         between them, and egress restricted so a
                         copy cannot leave by accident.
STEP 4 · IAM             Permission boundaries and SCPs that deny
                         cross-Region actions on the data stores.
                         This is where residency is enforced.
STEP 5 · compute         Unchanged. The execution model did not
                         move.
STEP 6 · load balancing  Unchanged, but Route 53 now routes on
                         geolocation rather than latency.
STEP 7 · data stores     No global tables, no cross-Region read
                         replicas. Two independent stacks.
STEP 8 · caching         Two caches. A shared one would be a
                         residency violation.
STEP 9 · queues          Per-Region queues, and any consumer that
                         previously read from both is now two.
STEP 10 · security       Customer-managed keys per Region, and
                         classification recorded per table.
STEP 11 · observability  Logs must stay in Region, so the central
                         log account becomes one per Region, with
                         only metrics aggregated globally.
STEP 12 · IaC            One module, two instantiations — the
                         change that makes all of the above
                         maintainable rather than duplicated.
STEP 13 · deployment     Two pipelines, or one that promotes to
                         both, and a decision about whether a
                         release can be live in one Region only.
STEP 14 · scaling        Unchanged in mechanism, doubled in cost.
STEP 15 · cost           Roughly double, and now with duplicated
                         fixed costs. Worth stating explicitly.
STEP 16 · recovery       No cross-Region failover for data. DR is
                         now within a Region, which changes the
                         achievable RTO and has to be re-agreed.

Fourteen of sixteen steps moved. That is what a requirement
change looks like in a derived design, and it is why step 1 is
step 1.
1
A residency requirement reads like a legal detail and is one of the most architecturally invasive things a design can be handed.
7
Steps 2 to 4 are where residency is actually enforced — everything downstream depends on the boundary being real rather than intended.
20
The two steps that do not move are worth naming too: knowing what a change does not affect is as much a part of the derivation as knowing what it does.

Why this works: Tracing a single requirement change through all sixteen steps is the clearest demonstration of why the order exists. The change enters at step 1 and lands hardest at steps 2 to 4, because those establish the boundaries everything else is placed inside. A design assembled in a different order would have to be unpicked rather than adjusted.

Starting at compute because that is the interesting part

Wrong

text
# "I would run this on ECS Fargate."
# — before accounts, network or IAM have been mentioned, and
#   before anyone has said how much traffic there is.

Better

text
# "Four numbers first. Then: separate production account, two
#  AZs, three subnet tiers, and these roles. Given that, ECS on
#  Fargate — because the workload holds connections, which
#  rules out Lambda, and nothing here needs the machine."

What you see: The design is defensible on its own terms and cannot be connected to the requirement, so every follow-up question is answered with a new assertion rather than with something already established.

Why: Compute is the most visible decision and the most dependent one: it is constrained by the network it sits in, the identities it assumes, and the throughput it must serve. Deciding it first means those three get retrofitted to justify it, which is exactly the direction that produces a design nobody can perturb.

Sixteen steps, in three widening bands

Each step is answered using the ones above it. An answer that could have been given before its prerequisites were known is a recalled answer.

  • Sixteen numbered steps grouped into three bands.
  • Band 1, where things live and who may touch them: 1 requirements, 2 account and environment strategy, 3 network topology, 4 IAM.
  • Band 2, what serves the traffic and holds the state: 5 compute choice, 6 load balancing, 7 data stores, 8 caching, 9 queues, 10 security.
  • Band 3, how it is run: 11 observability, 12 infrastructure as code, 13 deployment, 14 scaling, 15 cost, 16 failure recovery.
  • Arrows run downward between the bands, showing that each band is decided using the answers from the one above.
  • Footnote: the final step tests every earlier decision by asking what happens when it breaks.

Each step, what decides it, and the answer that shows the order was skipped

Each step, what decides it, and the answer that shows the order was skipped
#StepDecided byOut-of-order answer
1RequirementsAsking for rate, size, ratio, growth, and the agreed RPO/RTO"It needs to be scalable and reliable"
2Account and environment strategyHow strong the isolation between environments has to be"Everything in one account with tags"
3Network topologyWhich tiers exist, and what each may reach"It runs in a VPC"
4IAMWhat each workload must call, and across which boundaries"Least privilege" with no roles named
5Compute choiceExecution model first, then cost, between what is left"Lambda, because it is serverless"
6Load balancingLayer 7 or layer 4, and whether the tier is statelessA load balancer with sticky sessions holding real state
7Data storesThe access patterns, most frequent first"Postgres" before a single query was written down
8CachingWhat is read repeatedly, and what staleness is allowed"Add Redis" before the store or the miss cost was known
9QueuesWhat may finish after the response"It is asynchronous" with no consumer or backlog story
10SecurityClassification, encryption, secrets, audit trail"Everything is encrypted"
11ObservabilityWhich signals would actually make someone act"CloudWatch"
12Infrastructure as codeOne tool, and what it owns end to endA repository that covers most of the account
13DeploymentHow a bad release is reversed, migrations included"We deploy from the pipeline"
14ScalingThe metric that genuinely tracks load per tierCPU target tracking for a queue-driven worker
15CostThe top three drivers, by usage typeA monthly total with no breakdown
16Failure recoveryPer dependency: timeout, retries, degraded mode, owner"It retries"

Together

text
# Why 7 before 8, and 11 before 12 — the two orderings that
# get reversed most often, and what each reversal costs.

CACHING BEFORE DATA STORES
  Reversed : "We will use Redis for the hot reads, and Postgres
             behind it."
  What it hides: nobody has written down the access patterns, so
             nobody knows whether the store was the wrong shape.
             The cache makes an unsuitable store fast enough to
             ship, and the mismatch surfaces later as an
             invalidation bug nobody can reproduce.
  In order : list the queries, choose the store that serves them,
             THEN ask which reads repeat often enough to cache and
             how stale each may be. Sometimes the answer is none.

INFRASTRUCTURE AS CODE BEFORE OBSERVABILITY
  Reversed : the whole platform is codified, and dashboards and
             alarms are added by hand afterwards.
  What it hides: the signals are now outside the repository, so
             they drift, they are missing in the second Region,
             and the rebuild drill silently produces a platform
             with no alarms on it.
  In order : decide the four signals worth paging on, then codify
             them alongside the resources they watch. An alarm
             that is not in the repository does not exist in any
             environment you have not touched by hand.

Remember: Sixteen steps in three bands: where it lives and who may touch it, what serves traffic and holds state, and how it is watched, shipped, paid for and recovered. Data stores before caching, observability before infrastructure as code, and failure recovery last but never optional. If an answer could have been given before its prerequisites, it was recalled.

See also: the requirement you can answer · the design dimensions · reviewing across all pillars · the operations review

Advertisement