beginnerintermediateadvanced

AWS · ECS Deployment

ECS Deployment Architecture

The request chain, deployment health and rollback, and a full service with a database and cache

"Running" ≠ "healthy target"ECS and the load balancer measure different things
min% / max% set the room to work100/100 is a stuck deployment
Two failure detectors, not onecircuit breaker + CloudWatch alarms
Secrets by ARN, never by valuea task definition revision is immutable

1.Service → Task Definition → Task → Target Group → Load Balancer

A task definition describes what to run — image, CPU, memory, ports, roles, logging. A task is one running copy of it. A service keeps a desired number of tasks running and registers each one in a target group. A load balancer listener forwards matching requests to that target group. Every ECS debugging session is a walk along this chain.

Key features

A task definition is versioned: each change creates a new revision, and a service points at a specific revision
A service maintains the desired count, replaces failed tasks, and registers and deregisters tasks with its target group automatically
The target group's health check decides whether a task receives traffic; the container health check (if defined) decides whether ECS considers the container healthy
A listener rule on the load balancer selects a target group by host, path, or header — one load balancer can front many services
ECS resolves the image tag in the task definition to a digest and uses that digest for every other task in the service, so all tasks run identical images
If digest resolution fails three or more times the deployment continues without it, and with the circuit breaker enabled the deployment is failed and rolled back; on Fargate each task gets its own elastic network interface and security group

Walking the chain when requests fail

Walking the chain when requests fail
Symptom	Where to look	Common cause
503 from the load balancer	Target group health	No healthy targets — every task is failing the health check
Targets never become healthy	Health check path and task security group	Health check hits a path that requires auth, or the SG blocks the LB
Tasks start then stop repeatedly	Stopped-task reason + CloudWatch Logs	Crash on startup, missing secret, or wrong port
Some requests 404	Listener rules	A rule matches a path the service does not serve
Tasks stuck in PROVISIONING	Subnets and capacity	No available IPs, or no capacity in the capacity provider
Deployment never completes	minimumHealthyPercent / maximumPercent	The configuration leaves no room to start or stop a task

The request path, and where it breaks

arrives atforwards toif healthycontainerport

Load balancer

security group must allow the client

Listener + rule

host / path / header match

Target group

health check decides membership

Task

its security group must allow the load balancer

Container

listening on the mapped port

Signature

task definition (revision) → task → service (desired count) → target group (health) → listener rule → ALB

Task definition (versioned recipe) → task (one copy) → service (keeps N and registers them) → target group (health check decides traffic) → listener rule → load balancer. "Tasks running" and "targets healthy" are different facts; debug by walking the chain.

2.Deployment Health, Circuit Breakers, and Rollback

A rolling ECS deployment replaces tasks a few at a time, bounded by two percentages: how far below the desired count it may drop, and how far above it may go. Two independent mechanisms can then decide the deployment has failed — the circuit breaker, when tasks cannot start, and a CloudWatch alarm, when application metrics go bad — and both can roll back to the previous service revision.

Key features

`minimumHealthyPercent` is the lower limit on running healthy tasks during a deployment, as a percent of desired count, rounded up
`maximumPercent` is the upper limit on running tasks during a deployment, as a percent of desired count, rounded down
The two values must leave room to stop or start at least one task, or the deployment gets stuck and ECS emits a service event saying so
Circuit breaker and CloudWatch-alarm failure detection can be used separately or together; with both, the deployment fails as soon as either one's criteria are met
AWS's guidance: use the circuit breaker when tasks cannot start, and alarms when the failure shows up in application metrics — both support rolling back to the previous service revision
Unhealthy tasks are replaced using the same service revision they belong to, keeping old-revision failures independent of the new one, and secrets are resolved at task start by the execution role from Secrets Manager or Parameter Store rather than stored in the task definition

What the two percentages allow, at desired count 4

What the two percentages allow, at desired count 4
min% / max%	Scheduler may	Effect
50 / 100	Stop 2 first, then start 2	No extra capacity needed; capacity halves mid-deploy
100 / 200	Start 4 first, then stop 4	Full capacity throughout; needs double capacity briefly
100 / 150	Start 2, stop 2, repeat	Full capacity, moderate headroom — a common default shape
100 / 100	Nothing	Deployment is stuck; ECS emits a service event

Where each piece of configuration belongs

Where each piece of configuration belongs
Concern	Where it is set	Note
Pull the image, write logs, read secrets	Task execution role	Used by the ECS agent, not by your code
Call AWS APIs from the application	Task role	Used by your code — keep these two separate
Secret values	`secrets` in the container definition	Resolved at task start from Secrets Manager / Parameter Store
Log destination	`logConfiguration` (awslogs driver)	One log group per service, streams per task
Scale on load	Service auto scaling target tracking	Typically ALB request count per target, or CPU

A rolling deployment with both failure detectors on

Scheduler starts new tasks

Bounded above by maximumPercent

New tasks register with the target group

Health check decides when they take traffic

Old tasks drain and stop

Bounded below by minimumHealthyPercent

Circuit breaker watches task starts

Repeated start failures fail the deployment

CloudWatch alarms watch behaviour

A breaching alarm fails the deployment too

Rollback to the previous revision

Either detector can trigger it

Signature

minimumHealthyPercent / maximumPercent · circuit breaker (tasks cannot start) + alarms (tasks misbehave) → rollback

minimumHealthyPercent and maximumPercent set the room the scheduler has; leave enough to start or stop one task. Enable both failure detectors — circuit breaker for tasks that cannot start, CloudWatch alarms for tasks that start and misbehave — with rollback on. Secrets go in `secrets`, not `environment`; execution role and task role stay separate.

3.Deploying a Web Service with a Managed Database and Redis

The standard shape: an internet-facing Application Load Balancer in public subnets, ECS tasks in private subnets, RDS and ElastiCache in private subnets with no route to the internet, secrets in Secrets Manager, and logs in CloudWatch Logs. Security groups reference each other rather than CIDR ranges, so the rules describe the architecture.

Key features

Only the load balancer sits in public subnets — tasks, RDS, and ElastiCache all sit in private subnets with no inbound internet path
Security groups reference other security groups as sources, so a rule reads "from the task security group" rather than a CIDR that drifts
Outbound access from private tasks (pulling images, reaching Secrets Manager) goes through a NAT gateway, or through VPC endpoints to avoid NAT data charges
Secrets Manager holds the database URL and any API keys; the task execution role resolves them at task start into the container environment
Migrations run as a one-off ECS task in the deployment pipeline, not on container startup where every task and every scale-out would race
Scale the service on ALB request count per target rather than CPU — it is the signal users experience — and treat ElastiCache as a cache, since a design that breaks when it is emptied has a correctness bug, not a performance one

The security group rules, written out

The security group rules, written out
Group	Inbound	Outbound
sg-alb	443 from 0.0.0.0/0	To sg-task on the container port
sg-task	Container port from sg-alb	To sg-db 5432, sg-cache 6379, and 443 for AWS APIs
sg-db	5432 from sg-task	None required
sg-cache	6379 from sg-task	None required

Three tiers, three security groups

Public subnets

Application Load Balancer

sg-alb: 443 from 0.0.0.0/0

NAT gateway

egress only

Private subnets — compute

ECS tasks

sg-task: 8000 from sg-alb

Private subnets — data

RDS PostgreSQL

sg-db: 5432 from sg-task

ElastiCache

sg-cache: 6379 from sg-task

Signature

ALB (public) → tasks (private) → RDS + ElastiCache (private) · SGs reference SGs · secrets by ARN · migrate as a task

ALB in public subnets, tasks and data stores in private ones, security groups referencing each other. Build once and pin the digest, migrate as a separate task with additive changes, resolve secrets by ARN at task start, and deploy with both failure detectors and rollback enabled.

Spotted a mistake on this page? Tell us.

Read Sheet