beginnerintermediateadvanced

AWS · Section 41

CI/CD on AWS

Level: intermediate
Read: 35 min
Concepts: 4

A pipeline is a series of stages — source, build, test, deploy, with approval gates where a human decision is required — and an artifact that flows through all of them. The rule that makes lower environments meaningful is building that artifact once and promoting it by digest, so "it passed in staging" is a statement about the exact bytes now in production. On AWS the work splits across CodeBuild (run the build), CodePipeline (orchestrate stages, artifacts, approvals, and stage rollback) and CodeDeploy (shift traffic and roll back), and a common, defensible arrangement is to build and test in GitHub Actions or GitLab CI — authenticating via OIDC rather than stored keys — while deploying with CodeDeploy or CloudFormation. The deployment strategy then decides what serves traffic during the change: rolling and canary run both versions at once and therefore require backward compatibility, blue/green guarantees a single version serves and makes rollback a reroute, and feature flags separate deploying code from enabling behaviour. All Lambda and ECS CodeDeploy deployments are blue/green, with canary, linear, or all-at-once traffic shifting. Finally, database migrations belong inside this design rather than beside it: code rolls back and data does not, so schema changes are split expand/contract across releases and run as their own pipeline step.

This section

AWS CodePipeline tutorial | Build a CI/CD Pipeline on AWSBlock Explorer

What is true here

Build the artifact once and promote it by digest through every stage.
CodeBuild runs, CodePipeline orchestrates, CodeDeploy shifts traffic and rolls back.
Rolling and canary mean two versions serve at once; blue/green guarantees one does.
A canary needs an alarm judging it, or it is only a slower deploy.
Migrations are expand/contract, run as a pipeline step, and planned with their own rollback story.

What you will be able to do

Design a pipeline whose stages share one immutable artifact, with an approval gate and a rollback path
Split CI/CD responsibilities between AWS services and an existing source-control CI, using OIDC for credentials
Choose a deployment strategy from what the change requires, not just from risk appetite
Wire alarms into a canary so a bad version rolls back automatically
Sequence a schema change so every intermediate state works for both deployed versions

From a commit to a reversible production change

Stages, artifacts, rollback

CodeBuild / CodePipeline / CodeDeploy

Rolling, blue/green, canary, flags

Migrations designed with the deploy

CI/CD on AWS

Pipeline stages and artifact promotion, the AWS services and their alternatives, deployment strategies, and migrations as part of the deployment.

Pipeline Stages, Artifacts, and Rollback

coreintermediate

A pipeline describes how a code change reaches production. It is a series of stages — source, build, test, deploy — and each stage contains actions that operate on artifacts: the source tree, the built package, the deployment definition. The build stage produces an artifact once, and every later stage deploys that same artifact rather than rebuilding it.

Think of it as

A pipeline is an assembly line, and the artifact is the thing moving along it. The single most important rule follows from that image: the object tested in staging must be the same object deployed to production. Rebuilding between stages is putting a different item on the belt and hoping it is identical.

What we're doing: Understand why "it worked in staging" is only meaningful when the artifact is shared.

rebuild-vs-promote.txttext

Rebuild-per-stage pipeline:
  staging:    docker build ... -> image A (npm installed on Tuesday)
  production: docker build ... -> image B (npm installed on Thursday,
                                  transitive dependency bumped)
  Staging tested A. Production runs B. The test proved nothing about
  what is now serving traffic.

Promote-the-artifact pipeline:
  build once -> sha256:9f2c...
  staging and production both deploy sha256:9f2c...
  Staging tested exactly the bytes now in production.

2: Nothing here looks wrong. The Dockerfile is the same, the commit is the same — only the resolved dependency tree differs, which is exactly the class of difference nobody notices.
8: Building once and promoting by digest makes "tested in staging" a statement about the production artifact rather than about a sibling of it.

Why this works: The value of a lower environment comes entirely from it running the same thing production will. A pipeline that rebuilds per stage keeps the ceremony of testing and loses the guarantee, and the resulting failures are the hardest kind to reproduce.

Deploying by mutable tag instead of by digest

Wrong

text

# Production task definition references api:latest

Better

text

# Production task definition references api@sha256:9f2c…
# (or an immutable, commit-pinned tag)

What you see: Two tasks in the same service run different code, because one started before the tag was repointed and one after — and the service reports healthy throughout.

Why: A tag is a mutable pointer. Anything that pulls by tag resolves it at pull time, so a scale-out or a task replacement hours later silently picks up whatever the tag means then. A digest names specific bytes and cannot drift.

One artifact, many stages

Source

commit or tag triggers the pipeline

Build

produces the artifact — once

Test

against that artifact

Deploy staging

same artifact

Approval

human gate

Deploy production

same artifact again

What belongs in each stage

What belongs in each stage
Stage	Produces	Fails when
Source	The source revision (commit or object version)	The trigger filter does not match, or credentials expired
Build	The immutable artifact — image digest, zip, template	Compilation, dependency resolution, or a lint gate fails
Test	A pass/fail verdict on that artifact	A test fails, or coverage/security gates are not met
Deploy (staging)	A running environment on that artifact	Health checks fail, or a migration errors
Approval	A recorded human decision	It times out, or someone rejects it
Deploy (production)	The same artifact, live	Health checks fail — which should trigger rollback

Together

text

# Promote by digest, never by mutable tag
Build:      docker build -t $ECR/api:$GIT_SHA .
            docker push $ECR/api:$GIT_SHA
Staging:    deploy image $ECR/api@sha256:9f2c...
Production: deploy image $ECR/api@sha256:9f2c...   # identical bytes

Remember: Source → build → test → deploy → approval → deploy, with one artifact flowing through all of it. Build once, promote by digest, and make rollback a configured path rather than a forward fix written during an incident.

CodeBuild, CodePipeline, CodeDeploy — and the Alternatives

standardintermediate

AWS splits CI/CD into three services. CodeBuild runs build and test commands in a managed container. CodePipeline is the orchestration — the stages, the artifact hand-offs, the approvals. CodeDeploy handles the deployment itself, including blue/green traffic shifting on Lambda and ECS. GitHub Actions and GitLab CI cover the first two and often hand deployment to CodeDeploy or to the AWS CLI.

Think of it as

CodeBuild is the worker, CodePipeline is the foreman, CodeDeploy is the crew that swaps the running thing over. Most teams already have a worker and a foreman in their source-control platform, and what they genuinely need from AWS is the crew — the part that knows how to shift traffic and roll back.

text

# The three CodeDeploy compute platforms, and what a deployment means on each
EC2/On-Premises -> in-place rolling update, or blue/green with new instances
AWS Lambda      -> shift traffic between two versions of the same function
Amazon ECS      -> shift traffic from the original task set to a replacement

Storing long-lived AWS access keys as CI secrets

Wrong

text

# Repository secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

Better

text

# OIDC federation: the CI provider presents a token, IAM trusts it for a
# specific repository and branch, and issues short-lived credentials

What you see: A key leaked through a fork, a log line, or a compromised action stays valid until someone notices and rotates it — and rotation means updating every repository that holds a copy.

Why: A long-lived key is a bearer credential with no expiry and no binding to who is using it. OIDC federation issues short-lived credentials tied to a specific repository and workflow, so a leaked token is useless within minutes and the trust policy states exactly which repository may assume the role.

Who does what

Who does what
Job	AWS service	Common alternative
Run build and test commands	CodeBuild	GitHub Actions runner, GitLab CI runner
Orchestrate stages and approvals	CodePipeline	Workflow file with environments and required reviewers
Shift traffic and roll back	CodeDeploy	Rarely replaced — this is the piece worth keeping
Provision infrastructure	CloudFormation / CDK / Terraform	The same tools, invoked from any runner
Store built images	ECR	Any registry, but ECR keeps IAM as the auth model

Together

yaml

# GitHub Actions → AWS without stored keys: OIDC into a role
permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111122223333:role/github-deploy
      aws-region: eu-west-1

Remember: CodeBuild = run it, CodePipeline = orchestrate it, CodeDeploy = shift traffic and roll back. Building and testing in GitHub Actions or GitLab CI while deploying with CodeDeploy or CloudFormation is a normal, defensible split — as long as the CI system authenticates via OIDC rather than stored keys.

Rolling, Blue/Green, Canary, Immutable, Feature-Flagged

coreintermediate

Every deployment strategy is a different answer to one question: while the new version is going out, what is serving traffic? Rolling replaces instances in batches, so both versions run at once. Blue/green stands the new version up beside the old and switches traffic. Canary sends a small slice of traffic first. Immutable replaces rather than updates. Feature flags separate deploying the code from turning the behaviour on.

Think of it as

Rolling is repainting a bridge lane by lane while cars keep crossing. Blue/green is building a second bridge and redirecting traffic. Canary is opening the new bridge to a few cars first. Each costs more than the last and refunds you in how quickly and cleanly you can go back.

What we're doing: Pick a strategy for a change that cannot be made backward compatible.

strategy-choice.txttext

Change: the cache serialization format changes. v1 cannot read what v2
writes, and vice versa.

Rolling: v1 and v2 run side by side for several minutes, sharing one
cache. Both break on each other's entries. Not viable.

Canary: same problem, for longer. Not viable.

Blue/green with a separate cache namespace per version: only one version
serves traffic at a time, and each reads only its own entries. Viable.

Better still: make it backward compatible (versioned keys, dual-write)
and then any strategy works.

1: The constraint is not the deployment tool. It is whether two versions of the code can coexist against shared state.
5: This is the failure people are surprised by, because rolling deployment is the default in most orchestrators and the incompatibility is in the data, not the code.
9: Blue/green is the strategy that permits a non-backward-compatible change, because it guarantees a single version serves at any moment.

Why this works: Strategy choice is usually presented as a risk/cost trade-off, and there is a harder constraint underneath it: rolling and canary both require the two versions to coexist. When they cannot, blue/green is not the safer option, it is the only working one.

Canary deploying without a metric that judges the canary

Wrong

text

# Shift 10%, wait 10 minutes, shift the rest — nobody is watching

Better

text

# Wire the deployment to CloudWatch alarms on error rate and p99 latency
# so a breaching alarm rolls the deployment back automatically

What you see: The canary period passes uneventfully, the deployment completes, and the failure is reported by customers twenty minutes later — the 10% window collected the evidence and nothing read it.

Why: A canary is a measurement, not a waiting period. Without an alarm bound to the deployment, the gradual shift only slows down the arrival of a bad version rather than preventing it, and costs deploy time for nothing.

Two versions at once, or not

Rolling / canary — both versions live

+Cheapest in capacity: no second environment
+Requires backward-compatible changes
+Rollback means another rolling pass
+Canary limits exposure by traffic share

Blue/green — one version serves

−Needs capacity for two environments briefly
−Test the replacement before any traffic reaches it
−Rollback is rerouting traffic back
−The original can be kept running for a bake period

Choosing a strategy

Choosing a strategy
Strategy	Protects against	Costs
Rolling (in-place)	Total outage during deploy	Both versions live; slow rollback
Blue/green	A bad version reaching users at all	Double capacity during the switch
Canary	A failure that only shows under real traffic	Longer deploys; needs good metrics to judge the canary
Linear	Sudden full exposure	Longest deploy window of the three
Immutable	Drift and half-updated hosts	A new image per release
Feature flag	A behaviour change, independent of the deploy	Flag lifecycle management; dead flags accumulate

Together

text

# ECS/Lambda traffic shifting configurations, by shape
canary   : 10% now, the remaining 90% after N minutes
linear   : 10% every N minutes until 100%
all-at-once: 100% immediately (blue/green, but no gradual exposure)

Remember: Rolling and canary run two versions at once, so the change must be backward compatible. Blue/green guarantees one version serves and makes rollback a reroute. Canary needs an alarm judging it. Immutable replaces rather than updates. Feature flags separate deploy from release — and must be removed.

Database Migrations as Part of the Deployment

coreadvanced

Application code can be rolled back. A migration that dropped a column cannot. That asymmetry is why schema changes have to be designed alongside the deployment strategy rather than bolted onto it: during any rolling or canary deployment, the old and new code both talk to one schema, so the schema has to satisfy both at once.

Think of it as

Deploying code is reversible; changing data is not. So schema changes get split in two: first an additive change that both versions tolerate, and later — after the old version is gone for good — the destructive one. The expand/contract pattern is just that split, written down.

What we're doing: See how a one-line rename becomes an outage during an ordinary rolling deploy.

rename-during-rolling.txttext

Migration in the deploy step: ALTER TABLE orders RENAME COLUMN
total TO total_cents;

At that moment 6 of 10 tasks are still running the old version. They
issue SELECT total FROM orders and get an error.

40% of requests fail until the rolling deploy finishes — and rolling
back the code does not help, because the column is already renamed.

3: The migration is correct in isolation and correct for the new code. It is only wrong in the presence of the version it is replacing.
6: This is the asymmetry: the code rollback is one command, and it restores a version that the schema no longer supports.

Why this works: Rolling and canary deployments are defined by both versions running at once, and a migration that only the new version tolerates contradicts that. The fix is not a better deployment tool — it is splitting the schema change so that every intermediate state is valid for both versions.

Running migrations on application startup

Wrong

text

# entrypoint.sh: run migrations, then start the server

Better

text

# A separate pipeline step (an ECS run-task or a CodeBuild action) runs
# migrations once, before the deploy step that replaces tasks

What you see: Ten tasks start simultaneously and all ten attempt the migration; some fail on a lock, the service flaps, and the first successful start is a race.

Why: Migrations are a once-per-release operation and container starts are a many-times-per-release event — including scale-out and task replacement long after the deploy. Coupling them means the schema changes at moments nobody planned, under concurrency nobody designed for.

Expand/contract across three releases

Release 1 — expand

Add the new nullable column. Old code ignores it; nothing breaks.

Release 2 — dual write

New code writes both columns and still reads the old one. Backfill in the background.

Release 3 — switch reads

New code reads the new column. The old column is now unused but still present.

Release 4 — contract

Drop the old column, once no deployed version references it.

Migration shapes and their deployment risk

Migration shapes and their deployment risk
Change	Safe during a rolling deploy?	Handling
Add a nullable column	Yes	Old code ignores it
Add a column with a default	Usually	Check the engine's rewrite behaviour on a large table
Add an index	Yes, if built concurrently	A blocking index build locks writes
Rename a column	No	Expand/contract: add, dual-write, switch, drop
Drop a column	No	Only after no deployed version references it
Narrow a type or add NOT NULL	No	Backfill first, constrain in a later release

Together

text

# The rename, done safely, across releases
R1: ALTER TABLE orders ADD COLUMN total_cents integer;
R2: app writes total AND total_cents; backfill total_cents
R3: app reads total_cents
R4: ALTER TABLE orders DROP COLUMN total;

Remember: Two versions share one database during any rolling or canary deploy, so every schema state must satisfy both. Expand (additive), dual-write, switch reads, contract — across separate releases. Run migrations as their own pipeline step, and never ship an irreversible migration alongside the code that needs it.