beginnerintermediateadvanced

AWS · Section 43

ECS Deployment Architecture

Level: advanced
Read: 30 min
Concepts: 3

Every ECS debugging session is a walk along one chain: a versioned task definition describes what to run, a task is one copy of it, a service keeps the desired number running and registers each with a target group, and a listener rule forwards matching requests there. "Tasks running" and "targets healthy" are separate facts, which is why a service can look perfectly healthy while the load balancer returns 503. Deployments are bounded by two percentages — how far below the desired count the scheduler may drop and how far above it may go — and must leave room to start or stop at least one task or the deployment sticks. Two independent failure detectors then decide whether a rollout is going wrong: the deployment circuit breaker, for tasks that cannot start, and CloudWatch alarms, for tasks that start and misbehave; both support rolling back to the previous service revision, and used together the deployment fails as soon as either one's criteria are met. Assembled, the standard architecture is an ALB in public subnets, tasks and data stores in private ones, security groups referencing each other rather than CIDRs, secrets resolved by ARN at task start, migrations run as a separate one-off task, and logs going to a per-service CloudWatch log group.

This section

AWS ECS Full Tutorial | Learn EC2, EKS, Fargate, Load Balancers and Docker Deployment with DemoKodeKloud

What is true here

Task definition → task → service → target group → listener → ALB; debug by walking the chain.
minimumHealthyPercent and maximumPercent must leave room to start or stop a task.
Circuit breaker covers tasks that cannot start; CloudWatch alarms cover tasks that start and misbehave.
Secrets go in secrets (an ARN), not environment — task definition revisions are immutable and readable.
ALB public, everything else private, with security groups referencing security groups.

What you will be able to do

Trace a request from the load balancer to the container and locate where it fails
Set deployment percentages that neither halve capacity nor deadlock the scheduler
Enable both failure detectors with rollback, and explain which failure each one catches
Inject secrets safely and keep the execution role and task role separate
Lay out a full service with a managed database and cache, including migrations in the pipeline

From the request chain to a full deployment

Service → task → target group → ALB

Health, circuit breakers, rollback

A service with a database and a cache

ECS Deployment Architecture

The chain from load balancer to container, the mechanisms that make a deployment safe, and the full three-tier architecture.

Service → Task Definition → Task → Target Group → Load Balancer

coreintermediate

A task definition describes what to run — image, CPU, memory, ports, roles, logging. A task is one running copy of it. A service keeps a desired number of tasks running and registers each one in a target group. A load balancer listener forwards matching requests to that target group. Every ECS debugging session is a walk along this chain.

Think of it as

The task definition is a recipe, a task is a dish made from it, the service is the kitchen that keeps N dishes on the pass, and the target group is the pass itself — the list of dishes the waiter is allowed to pick up. A request that fails has stopped somewhere specific along that line.

What we're doing: Diagnose a service that deploys "successfully" but serves 503s.

walk-the-chain.txttext

ALB returns 503. Service says 4/4 tasks running.

1. Target group: 0 of 4 targets healthy. So the tasks are running but
   the load balancer will not send them traffic.

2. Health check: path /health, but the application mounts it at
   /healthz. Every check returns 404, which is not a healthy code.

3. Fix the health check path. Targets go healthy within one interval and
   the 503s stop.

"Tasks running" and "targets healthy" are two different facts.

3: This is the split that confuses people: ECS keeps tasks running, the target group decides whether they receive traffic, and neither knows about the other's definition of healthy.
6: A health check hitting a path that redirects, requires authentication, or does not exist is the single most common cause of permanently unhealthy targets.
9: The deployment itself was never wrong. Nothing in the ECS console was red.

Why this works: ECS reports on tasks and the load balancer reports on targets. A service can be perfectly healthy by one measure and serving nothing by the other, so the debugging habit that pays is walking the whole chain rather than trusting the first green indicator.

Pointing the target group health check at a path that touches the database

Wrong

text

# Health check path: /health — which runs a SELECT to prove the DB is up

Better

text

# Liveness (target group): /healthz — process is up, no dependencies
# Readiness/dependency checks: a separate endpoint, monitored not
# load-balanced

What you see: A brief database blip marks every task unhealthy at once, the load balancer removes them all, and a recoverable database hiccup becomes a total outage.

Why: The health check decides whether a task receives traffic. Making it depend on a shared downstream turns any downstream problem into a fleet-wide removal, and the tasks were capable of serving cached or degraded responses the whole time.

The request path, and where it breaks

Load balancer

security group must allow the client

Listener + rule

host / path / header match

Target group

health check decides membership

Task

its security group must allow the load balancer

Container

listening on the mapped port

Walking the chain when requests fail

Walking the chain when requests fail
Symptom	Where to look	Common cause
503 from the load balancer	Target group health	No healthy targets — every task is failing the health check
Targets never become healthy	Health check path and task security group	Health check hits a path that requires auth, or the SG blocks the LB
Tasks start then stop repeatedly	Stopped-task reason + CloudWatch Logs	Crash on startup, missing secret, or wrong port
Some requests 404	Listener rules	A rule matches a path the service does not serve
Tasks stuck in PROVISIONING	Subnets and capacity	No available IPs, or no capacity in the capacity provider
Deployment never completes	minimumHealthyPercent / maximumPercent	The configuration leaves no room to start or stop a task

Together

text

# Why did the last tasks stop?
aws ecs describe-tasks --cluster prod --tasks $(aws ecs list-tasks \
  --cluster prod --service-name api --desired-status STOPPED \
  --query 'taskArns[0]' --output text) \
  --query 'tasks[].{reason:stoppedReason,containers:containers[].reason}'

Remember: Task definition (versioned recipe) → task (one copy) → service (keeps N and registers them) → target group (health check decides traffic) → listener rule → load balancer. "Tasks running" and "targets healthy" are different facts; debug by walking the chain.

Deployment Health, Circuit Breakers, and Rollback

coreadvanced

A rolling ECS deployment replaces tasks a few at a time, bounded by two percentages: how far below the desired count it may drop, and how far above it may go. Two independent mechanisms can then decide the deployment has failed — the circuit breaker, when tasks cannot start, and a CloudWatch alarm, when application metrics go bad — and both can roll back to the previous service revision.

Think of it as

The two percentages set how much room the scheduler has to work in; the circuit breaker and the alarms are the two ways it can be told to stop. One watches whether the new tasks come up at all, the other watches whether they behave once they do.

What we're doing: Stop a bad deployment automatically instead of discovering it from customer reports.

circuit-breaker-vs-alarm.txttext

Failure A — the new image crashes on startup (missing environment
variable). Tasks never reach a healthy state.
  -> Circuit breaker detects repeated start failures, fails the
     deployment and rolls back to the previous revision.

Failure B — the new image starts fine and returns 500 on one endpoint.
Tasks are healthy; the target group is happy.
  -> The circuit breaker sees nothing wrong. A CloudWatch alarm on the
     target group's 5xx rate is what fails this deployment.

1: The circuit breaker is about the task lifecycle: can this thing start and stay up.
6: The alarm is about behaviour. A deployment with only the circuit breaker enabled will happily complete a rollout of code that starts perfectly and serves errors.

Why this works: The two detectors cover disjoint failure modes, which is why AWS documents them as usable together — with both on, the deployment fails as soon as either one's criteria are met. Enabling only the circuit breaker is the common half-measure.

Passing secrets as plain environment variables in the task definition

Wrong

json

"environment": [{ "name": "DATABASE_URL", "value": "postgres://user:pa55w0rd@..." }]

Better

json

"secrets": [{ "name": "DATABASE_URL", "valueFrom": "arn:aws:secretsmanager:...:secret:prod/db-AbCdEf" }]

What you see: The password is visible to anyone who can call `describe-task-definition`, appears in CloudFormation templates and CI logs, and is captured in every task definition revision permanently.

Why: A task definition is metadata anyone with read access to ECS can retrieve, and its revisions are immutable — so a secret placed there cannot be removed, only superseded. The `secrets` field stores an ARN and has the execution role resolve the value at task start, which keeps the value out of the definition entirely.

A rolling deployment with both failure detectors on

Scheduler starts new tasks

Bounded above by maximumPercent

New tasks register with the target group

Health check decides when they take traffic

Old tasks drain and stop

Bounded below by minimumHealthyPercent

Circuit breaker watches task starts

Repeated start failures fail the deployment

CloudWatch alarms watch behaviour

A breaching alarm fails the deployment too

Rollback to the previous revision

Either detector can trigger it

What the two percentages allow, at desired count 4

What the two percentages allow, at desired count 4
min% / max%	Scheduler may	Effect
50 / 100	Stop 2 first, then start 2	No extra capacity needed; capacity halves mid-deploy
100 / 200	Start 4 first, then stop 4	Full capacity throughout; needs double capacity briefly
100 / 150	Start 2, stop 2, repeat	Full capacity, moderate headroom — a common default shape
100 / 100	Nothing	Deployment is stuck; ECS emits a service event

Together

text

aws ecs update-service --cluster prod --service api \
  --task-definition api:47 \
  --deployment-configuration \
    'minimumHealthyPercent=100,maximumPercent=200,\
deploymentCircuitBreaker={enable=true,rollback=true}'

Where each piece of configuration belongs

Where each piece of configuration belongs
Concern	Where it is set	Note
Pull the image, write logs, read secrets	Task execution role	Used by the ECS agent, not by your code
Call AWS APIs from the application	Task role	Used by your code — keep these two separate
Secret values	`secrets` in the container definition	Resolved at task start from Secrets Manager / Parameter Store
Log destination	`logConfiguration` (awslogs driver)	One log group per service, streams per task
Scale on load	Service auto scaling target tracking	Typically ALB request count per target, or CPU

Together

json

{
  "secrets": [
    { "name": "DATABASE_URL",
      "valueFrom": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/db-AbCdEf" }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": { "awslogs-group": "/aws/ecs/api", "awslogs-region": "eu-west-1",
                 "awslogs-stream-prefix": "task" }
  }
}

Remember: minimumHealthyPercent and maximumPercent set the room the scheduler has; leave enough to start or stop one task. Enable both failure detectors — circuit breaker for tasks that cannot start, CloudWatch alarms for tasks that start and misbehave — with rollback on. Secrets go in `secrets`, not `environment`; execution role and task role stay separate.

Deploying a Web Service with a Managed Database and Redis

coreadvanced

The standard shape: an internet-facing Application Load Balancer in public subnets, ECS tasks in private subnets, RDS and ElastiCache in private subnets with no route to the internet, secrets in Secrets Manager, and logs in CloudWatch Logs. Security groups reference each other rather than CIDR ranges, so the rules describe the architecture.

Think of it as

Three tiers and three security groups. The load balancer accepts the internet; the tasks accept only the load balancer; the data stores accept only the tasks. Each layer names the layer above it as its source, which makes the diagram and the rules the same document.

What we're doing: Walk the deployment of a FastAPI service from pipeline to serving traffic.

end-to-end-deploy.txttext

1. Build once: docker build, push to ECR, capture the digest.

2. Migrate: run-task with the same image, command ["alembic","upgrade",
   "head"], one task, wait for exit code 0. Additive changes only, so the
   currently running version keeps working.

3. Register a task definition revision pinned to the digest, with
   secrets: DATABASE_URL and REDIS_URL from Secrets Manager, and an
   awslogs logConfiguration.

4. update-service to that revision, minimumHealthyPercent=100,
   maximumPercent=200, circuit breaker with rollback enabled, plus a
   CloudWatch alarm on target 5xx wired into the deployment.

5. New tasks register with the target group, pass /healthz, take traffic;
   old tasks drain and stop.

1: One build. Everything downstream refers to the digest, so staging and production run identical bytes.
5: Migrations before the deploy, and additive only — the old version is still serving while this runs.
9: Secrets by ARN, never by value. The task definition revision is safe to read and safe to keep.
12: Both failure detectors on. The circuit breaker catches a crash-on-start; the alarm catches code that starts fine and serves errors.

Why this works: Each step exists because of a specific failure it prevents: rebuild drift, a migration that breaks the running version, a secret in metadata, and a bad deployment that nobody stops. The shape is the same for Django, FastAPI, or Node — only the migration command changes.

Putting the ECS tasks in public subnets to make image pulls work

Wrong

text

# Tasks in public subnets with assignPublicIp=ENABLED, "because
# otherwise they cannot pull from ECR"

Better

text

# Tasks in private subnets, with a NAT gateway for egress — or VPC
# endpoints for ECR, S3, Logs, and Secrets Manager to avoid NAT charges

What you see: Every task has a public IP address. Nothing is exposed today because the security group is closed, and one over-permissive rule later, the tasks are directly reachable from the internet.

Why: A public IP means reachability depends entirely on the security group being correct forever. Private subnets remove the possibility rather than filtering it, and the egress problem the public placement was solving has two clean answers: NAT for general egress, or VPC endpoints for the specific AWS services involved.

Three tiers, three security groups

Public subnets

Application Load Balancer

sg-alb: 443 from 0.0.0.0/0

NAT gateway

egress only

Private subnets — compute

ECS tasks

sg-task: 8000 from sg-alb

Private subnets — data

RDS PostgreSQL

sg-db: 5432 from sg-task

ElastiCache

sg-cache: 6379 from sg-task

The security group rules, written out

The security group rules, written out
Group	Inbound	Outbound
sg-alb	443 from 0.0.0.0/0	To sg-task on the container port
sg-task	Container port from sg-alb	To sg-db 5432, sg-cache 6379, and 443 for AWS APIs
sg-db	5432 from sg-task	None required
sg-cache	6379 from sg-task	None required

Together

text

# The rule that makes the architecture self-documenting
aws ec2 authorize-security-group-ingress \
  --group-id sg-db --protocol tcp --port 5432 \
  --source-group sg-task

Remember: ALB in public subnets, tasks and data stores in private ones, security groups referencing each other. Build once and pin the digest, migrate as a separate task with additive changes, resolve secrets by ARN at task start, and deploy with both failure detectors and rollback enabled.