Filter concepts by levelShowing all levels.

Django · Section 95

CI/CD

Level
advanced
Read
30 min
Concepts
3

The section draws a pipeline and then lists what makes one work. The ordering in that diagram is the first lesson: lint, type check, unit tests, integration tests, security scan, build, deploy, migrate, health check, smoke test — cheapest first, so a two-second failure never waits behind a five-minute one, and the security scan sits before the build so a known-vulnerable dependency never becomes an image. After the build, every stage acts on the artefact rather than on the source. GitHub Actions and GitLab CI are the same shape in two vocabularies — YAML in the repository, jobs made of steps, a fresh machine per run — so what transfers is the structure and the discipline. Three properties are worth insisting on: every run starts clean, the environment is declared rather than discovered (a pinned interpreter, a pinned service image, a lock file), and the test environment is a real PostgreSQL service container. That last one matters more than it looks, because SQLite exercises your Python and almost none of your schema, so a suite on SQLite is green exactly when constraints, transactions or migrations are wrong. Two Django-specific gates belong in the list and are missing from most pipelines: `makemigrations --check --dry-run`, which fails when a model changed and nobody generated the migration, and `check --deploy`, which Django's checklist says to run "against your production settings file". Next, the credentials. A CI runner is the most privileged machine most teams own — it can read the repository, it holds deploy keys, and it runs third-party code on every push — so secrets belong in the masked store, injected only into the jobs that need them, with each job's token narrowed to what it does. Assume masking covers the literal string and nothing else: a secret leaves through a log line added while debugging, through a traceback with `DEBUG` on, through an `ENV` layer that is permanent and distributable, or through a fork's pull request running with the same variables. Dependency scanning belongs against the lock file, because that is the tree that ships, and the policy should be graded — fail on critical runtime findings, report development-only ones — since a gate that fails on everything is a gate that gets switched off. Finally, what happens to the artefact. Build once, tag by commit SHA, and promote that digest through staging into production; rebuilding per environment produces artefacts that can differ in ways no code diff shows, which removes the inference the pipeline exists to support. Keep images long enough to redeploy any release you might return to, put approvals on protected environments and record which digest was approved, and be clear-eyed about rollback: redeploying the previous image is fast and safe only when the migration that went with it was expand-shaped. A migration that dropped or renamed a column makes the previous version unrunnable, and the release plan should say "forward fix only" before the deploy rather than during the incident.

What is true here

  1. Gates run cheapest-first; after the build, everything acts on the artefact.
  2. A real database in CI is what makes the suite predictive of production.
  3. The runner holds secrets and runs third-party code — scope both per job.
  4. Scan the lock file, before the build, with a graded policy.
  5. Build once and promote; a rollback exists only if the schema still fits the old code.

What you will be able to do

  • Order a pipeline so failures arrive as early and as cheaply as possible
  • Add the two Django gates a passing test suite does not provide
  • Scope secrets and permissions per job, and name the ways one still escapes
  • Promote a single artefact through environments and plan a rollback that works
The roadmap's pipeline, with what each gate is protecting
nothing vulnerablebecomes an imagepromotethe digestsmoke passedsmoke failedonly if themigration expanded

Git push

a fresh machine, nothing cached unless you asked

Lint · type check

seconds — fails before anything expensive starts

Unit tests

logic, no services

Integration tests

real PostgreSQL and Redis service containers

Django gates

makemigrations --check · check --deploy (production settings)

Security scan

the lock file, before the build — graded policy

Build image once

tagged by commit SHA; the digest is the identity

Deploy staging

the same digest — never a rebuild

Approval

a named person, on a protected environment, for that digest

Deploy · migrate · health · smoke

expand-only migrations if a rollback must exist

Rollback

redeploy the previous digest — no build, no new decision

  • Git push — a fresh machine, nothing cached unless you asked
    • leads to Lint · type check
  • Lint · type check — seconds — fails before anything expensive starts
    • leads to Unit tests
  • Unit tests — logic, no services
    • leads to Integration tests
  • Integration tests — real PostgreSQL and Redis service containers
    • leads to Django gates
  • Django gates — makemigrations --check · check --deploy (production settings)
    • leads to Security scan
  • Security scan — the lock file, before the build — graded policy
    • leads to Build image once (nothing vulnerable becomes an image)
  • Build image once — tagged by commit SHA; the digest is the identity
    • leads to Deploy staging (promote the digest)
  • Deploy staging — the same digest — never a rebuild
    • leads to Approval (smoke passed)
  • Approval — a named person, on a protected environment, for that digest
    • leads to Deploy · migrate · health · smoke
  • Deploy · migrate · health · smoke — expand-only migrations if a rollback must exist
    • on error, leads to Rollback (smoke failed)
  • Rollback — redeploy the previous digest — no build, no new decision
    • leads to Deploy · migrate · health · smoke (only if the migration expanded)

The pipeline

Gate order, provider shape, and the environment the tests actually run against.

The pipeline, and the gates in the right order

coreintermediate

A pipeline is a fixed sequence that runs on every push: lint, type check, unit tests, integration tests, security scan, build, deploy, migrate, health check, smoke test. **GitHub Actions** and **GitLab CI** both describe it as YAML in your repository — jobs made of steps, running on a fresh machine each time. A **test environment** is what those tests run against: a real PostgreSQL and Redis started for the job, not mocks and not production.

Think of it as

The order in the roadmap's diagram is not decorative — it is cheapest-first, and that ordering is the whole economics of CI. Lint and type checks take seconds and catch a category of mistake outright, so they run before anything expensive. Unit tests run next because they need no services. Integration tests come after, because they need a database and a cache, which cost time to start. The security scan sits before the build so a known-vulnerable dependency stops the pipeline before you spend minutes producing an image with it inside. Then build once, and everything after that operates on the artefact rather than on the source. What makes this a *pipeline* rather than a script is that each gate can fail independently and tells you which class of problem you have: a lint failure and a smoke-test failure are the same red cross with completely different meanings. Both providers share the same shape — a YAML file in the repository, jobs that run on a fresh machine, steps inside a job that share a working directory — so the transferable knowledge is the structure and the discipline, not the syntax. Three properties are worth insisting on regardless of provider. First, every run starts clean: a fresh container, nothing cached unless you asked for it, which is what makes a green pipeline mean something. Second, the environment is declared, not discovered — a pinned Python version, a pinned PostgreSQL service, a lock file — so a pipeline that was green in March is green in September for the same commit. Third, the test environment is a real database. Django's test runner creates and destroys a test database, so pointing the job at a PostgreSQL service container gives you the same engine as production, which is the only way constraints, transactions and migrations are actually exercised. SQLite in CI against PostgreSQL in production is a well-worn way to discover a difference in a release rather than a test. Two Django-specific gates deserve a place in the list. `manage.py makemigrations --check --dry-run` fails when a model change has no migration, which is the most common way a deploy breaks with a green test suite. And `manage.py check --deploy` runs Django's own production checks; Django's checklist tells you to "run it against your production settings file", which in CI means setting `DJANGO_SETTINGS_MODULE` to the production module rather than the test one.

bash
python manage.py makemigrations --check --dry-run   # fails if a migration is missing

What we're doing: Write a Django CI job that runs against a real database and catches the two Django-specific failures a test suite does not.

.github/workflows/ci.ymlyaml
name: ci
on: [push, pull_request]

jobs:
  # Seconds, no services, and it fails the whole pipeline before anything
  # expensive starts. Kept as its own job so its result arrives even when
  # the test job is still running.
  static:
    runs-on: ubuntu-latest
    permissions: { contents: read }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install ruff mypy
      - run: ruff check .
      - run: mypy .

  test:
    runs-on: ubuntu-latest
    permissions: { contents: read }
    services:
      # The engine production uses. Django creates its test database
      # inside this container, so constraints and migrations are really
      # exercised — which SQLite would silently not do.
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: ci, POSTGRES_DB: ci }
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s --health-retries 10
        ports: ["5432:5432"]
    env:
      DATABASE_URL: postgres://postgres:ci@localhost:5432/ci
      DJANGO_SETTINGS_MODULE: config.settings.ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -e ".[dev]"

      # The gate a green test suite does not give you: a model changed
      # and nobody generated the migration. This fails the build now
      # rather than at "deploy -> migrate" on a Friday evening.
      - run: python manage.py makemigrations --check --dry-run

      # Django's own production checks. The checklist says to run it
      # "against your production settings file", so the settings module
      # is overridden here rather than reusing the CI one.
      - run: python manage.py check --deploy --fail-level WARNING
        env: { DJANGO_SETTINGS_MODULE: config.settings.production }

      - run: pytest -q --cov=. --cov-report=term-missing
5–10
Static checks as a separate job, so lint and type results arrive in parallel with the slower test job rather than serialising behind it. `permissions` is narrowed on both jobs.
22–31
A real PostgreSQL service with a health check, so the test step does not start against a database that is still initialising. `pg_isready` is the same probe the Compose file uses.
33–35
The database URL and settings module are job-level environment, which keeps every step consistent and makes the test environment a declared thing rather than an assumed one.
42–45
`makemigrations --check --dry-run` exits non-zero when a model change has no migration. This is the single most valuable Django-specific line in a CI file, because the test suite passes without it.
47–51
`check --deploy` is only meaningful against production settings — Django's checklist says so explicitly — so the module is overridden for this step alone. `--fail-level WARNING` makes it a gate rather than a note.

Why this works: Cheap checks fail fast and in parallel, tests run against the production database engine, and the two Django failures that a green test suite hides — a missing migration and an unsafe production setting — both stop the pipeline.

Testing against SQLite when production is PostgreSQL

Wrong

python
# config/settings/ci.py
DATABASES = {"default": {"ENGINE": "django.db.backends.sqlite3",
                         "NAME": ":memory:"}}   # fast, and not what you run

Better

python
DATABASES = {"default": dj_database_url.parse(os.environ["DATABASE_URL"])}
# a postgres:16 service container — the same engine as production

What you see: A migration that applies cleanly in CI and fails in production, a `UniqueConstraint` with a condition that is silently not enforced, or a `JSONField` query that works in one engine and not the other.

Why: Databases differ in exactly the places Django lets you be specific: constraint support, transactional DDL, locking behaviour, `select_for_update` semantics, JSON operators, and how migrations behave on a large table. A test suite on SQLite exercises your Python and very little of your schema, so it is green precisely when the schema is wrong. A service container costs a few seconds of start-up and removes an entire class of "worked in CI" incident. Where the suite is genuinely slow, the answer is to run integration tests as a separate job or to parallelise them — not to change the engine underneath them.

One CI job, part by part

jobs: test: runs-on: ubuntu-latest permissions: { contents: read } services: db: { image: postgres:16 } steps: - uses: actions/checkout@v4 - run: python -m pytest -q

jobs:

Jobs run in parallel by default — Independent jobs start together; `needs:` is what imposes order. Splitting lint and tests into separate jobs gives you both results at once instead of stopping at the first.

runs-on: ubuntu-latest

A fresh machine, every run — Nothing survives from the last run unless you cache it deliberately. That is what makes a green pipeline meaningful — and why "works on my machine" cannot happen here.

permissions: { contents: read }

The token this job gets — Narrow it explicitly. A job that only runs tests has no reason to be able to write to the repository, and a compromised dependency inherits exactly these permissions.

services:

Real backing services — A PostgreSQL container started for this job. Django creates and destroys its test database inside it, so constraints, transactions and migrations run against the engine production uses.

actions/checkout@v4

A pinned third-party step — A version, not a moving branch. An action runs arbitrary code in a context that holds your secrets, so what it resolves to should not change between runs without you choosing it.

python -m pytest -q

The step that is just a command — It runs the same thing you run locally. When the pipeline gets clever here — bespoke flags, CI-only paths — a green pipeline stops predicting a green machine.

  • Whole: jobs: test: runs-on: ubuntu-latest permissions: { contents: read } services: db: { image: postgres:16 } steps: - uses: actions/checkout@v4 - run: python -m pytest -q
  • jobs: — Jobs run in parallel by default: Independent jobs start together; `needs:` is what imposes order. Splitting lint and tests into separate jobs gives you both results at once instead of stopping at the first.
  • runs-on: ubuntu-latest — A fresh machine, every run: Nothing survives from the last run unless you cache it deliberately. That is what makes a green pipeline meaningful — and why "works on my machine" cannot happen here.
  • permissions: { contents: read } — The token this job gets: Narrow it explicitly. A job that only runs tests has no reason to be able to write to the repository, and a compromised dependency inherits exactly these permissions.
  • services: — Real backing services: A PostgreSQL container started for this job. Django creates and destroys its test database inside it, so constraints, transactions and migrations run against the engine production uses.
  • actions/checkout@v4 — A pinned third-party step: A version, not a moving branch. An action runs arbitrary code in a context that holds your secrets, so what it resolves to should not change between runs without you choosing it.
  • python -m pytest -q — The step that is just a command: It runs the same thing you run locally. When the pipeline gets clever here — bespoke flags, CI-only paths — a green pipeline stops predicting a green machine.

The roadmap's pipeline, with what each stage actually catches

The roadmap's pipeline, with what each stage actually catches
StageTypical durationThe failure it exists to catch
lintsecondsstyle and obvious errors, before anything costly runs
type checksecondsa signature that no longer matches its callers
unit testsa minute or twologic, with no services involved
integration testslonger — services startthe parts that only fail against a real database
security scanseconds to a minutea known-vulnerable dependency, **before** the build
build → imageminutesa build that does not reproduce, or a missing file
deploy → migrate → health → smokeminuteseverything that only exists in a real environment

Together

bash
ruff check . && mypy . && pytest -q -m "not integration" \
  && pytest -q -m integration && pip-audit && docker build -t app:$SHA .

GitHub Actions and GitLab CI — the same shape, two vocabularies

GitHub Actions and GitLab CI — the same shape, two vocabularies
ConceptGitHub ActionsGitLab CI
file`.github/workflows/ci.yml``.gitlab-ci.yml`
unit of worka `job`, made of `steps`a `job`, made of `script` lines
ordering`needs:``stage:` and `needs:`
services`services:` on the job`services:` on the job
reusable stepan `action` (`uses:`)an `include:` or a template
secrets`secrets.NAME`, masked in logsCI/CD variables, masked and protected

Together

yaml
# Actions
- uses: actions/checkout@v4
- run: pytest -q

# GitLab
script:
  - pytest -q

Remember: Run the gates cheapest-first — lint, types, unit, integration, security scan — so a two-second failure never waits behind a five-minute one, and build once so everything afterwards acts on the artefact. Both GitHub Actions and GitLab CI are the same shape: YAML in the repository, jobs of steps, a fresh machine each run. Test against a real PostgreSQL service container, because SQLite is green exactly when the schema is wrong. Install from a lock file so a red build means your change. And add the two Django gates a test suite does not give you: `makemigrations --check --dry-run`, and `check --deploy` run against the production settings module.

See also: secrets and dependency scanning · artifacts approvals and rollback · pytest django and fixtures · dependencies secrets and deployment checks

Advertisement

What the pipeline holds, and what it checks

Secrets on a privileged machine, and scanning the tree that really ships.

Secrets in the pipeline, and scanning what you depend on

coreadvanced

CI needs credentials — a registry login, a deploy key, sometimes a database password — and every one of them is handled by a machine that also runs code from your dependencies. So secrets live in the provider's masked store, are injected per job, and are never printed. **Dependency scanning** is the step that checks your locked dependency tree against known vulnerabilities, and it belongs *before* the build, so a vulnerable package never becomes an image.

Think of it as

A CI runner is the most privileged machine in most engineering organisations: it can read the repository, it holds deploy credentials, and it executes third-party code on every run — your dependencies, and the actions or templates the pipeline itself uses. Treating it that way changes four decisions. First, where secrets live. In the provider's store, masked in logs, and injected only into the jobs that need them — a test job needs no registry credentials, and a build job needs no production database password. Narrow the token the job runs with too, because a compromised dependency inherits exactly the permissions the job was given. Second, how they leak, because "we used the secrets store" is not the end of the story. A secret reaches the log the moment something prints it: an `echo` added while debugging, a `set -x` that expands every command, a stack trace with `DEBUG=True` rendering settings, a test that dumps the environment on failure. Masking helps with the exact string and not with a base64 of it or a fragment. It reaches an image the moment it appears in an `ENV` or a build argument, and an image layer is permanent and distributable. And it reaches strangers the moment a pull request from a fork can run with the same secrets, which is why pipelines distinguish trusted and untrusted triggers. Third, what to scan and when. Scanning is only meaningful against the resolved tree — the lock file — because that is what actually ships; scanning the manifest checks ranges rather than the versions installed. Put it before the build so a known-vulnerable package stops the pipeline early, and add a secret scanner alongside it, because a credential committed by accident is the most common serious finding in real repositories. Fourth, what a finding is worth. A scanner reports what a package *could* enable, not what your code does with it, so the useful policy is graded: fail the build on a critical or high vulnerability in a runtime dependency, warn on a development-only one, and require a written reason with an expiry date for anything you choose to accept. A pipeline that fails on everything gets a blanket ignore added within a fortnight, which is strictly worse than a policy someone still reads.

yaml
env:
  REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}   # on the push job only

What we're doing: Scan the resolved dependency tree and the repository for secrets, with a graded policy the team will not blanket-ignore.

.github/workflows/security.ymlyaml
name: security
on: [push, pull_request]

jobs:
  audit:
    runs-on: ubuntu-latest
    # No secrets are needed to scan, so none are made available. A
    # compromised scanner inherits exactly this: read access, nothing.
    permissions: { contents: read }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }

      # Scan the LOCK FILE. The manifest lists ranges; the lock file
      # lists the versions that will actually be installed and shipped,
      # which is the only tree a finding can be true about.
      - run: pip install pip-audit && pip-audit -r requirements.lock

      # Graded, not absolute. Runtime criticals stop the pipeline;
      # dev-only findings are reported and do not. A pipeline that fails
      # on everything gets a blanket ignore added within a fortnight.
      - run: pip-audit -r requirements-dev.lock || true

      # Secrets committed by accident are the most common serious
      # finding in real repositories, and the only fix is rotation —
      # so catching it at the pull request is worth a job of its own.
      - run: |
          pip install detect-secrets
          detect-secrets scan --baseline .secrets.baseline

      # Django's own production checks, as a security gate rather than a
      # deployment one: DEBUG, SECRET_KEY, ALLOWED_HOSTS, cookie flags.
      - run: python manage.py check --deploy --fail-level WARNING
        env:
          DJANGO_SETTINGS_MODULE: config.settings.production
          DJANGO_SECRET_KEY: not-a-real-key-only-to-satisfy-import
          # A placeholder, deliberately worthless: the check needs the
          # setting to exist, not to be the production value.
7–9
The job that scans needs no credentials at all, so it is given none. Scoping by job is what makes a compromised step in one job unable to reach another job's secrets.
15–18
The lock file is the tree that ships. Auditing the manifest audits ranges, which means a finding may not apply and a real vulnerability may not appear.
20–23
The graded half. Development-only dependencies do not run in production, so a finding there is information; making it fatal is how teams learn to bypass the whole job.
25–30
Secret scanning with a baseline, so known-and-accepted matches do not re-alarm while a genuinely new one still fails. A committed credential must be rotated — removing the commit does not un-disclose it.
32–38
`check --deploy` against production settings, with a deliberately worthless placeholder for the one setting that must merely exist. Passing the real key here would put a production secret in a scan job that needs none.

Why this works: The scanning jobs hold no credentials, findings are about the tree that actually ships, the policy distinguishes runtime from development, and Django's own production checks run as a gate.

Echoing a secret to debug the pipeline

Wrong

bash
- run: |
    set -x                       # expands every command, including secrets
    echo "DATABASE_URL=$DATABASE_URL"
    ./deploy.sh

Better

bash
- run: |
    test -n "$DATABASE_URL" || { echo "DATABASE_URL is unset"; exit 1; }
    ./deploy.sh                  # assert presence, never print the value

What you see: Nothing at all, immediately. The credential now sits in a build log that is retained for months, searchable, and readable by everyone with access to the repository — including anyone added to it later.

Why: Masking replaces the exact secret string in log output, which helps with a direct `echo` and not with the many ways a value gets transformed on the way out: base64, a URL-encoded fragment, a substring inside a traceback, or a value split across two lines. `set -x` is worse than a single `echo`, because it expands every command for the rest of the script, including the ones you did not write. When a pipeline needs debugging, assert the *presence* of a variable and print its length or a hash, never its value — and if a secret does reach a log, the only real remedy is rotation, because you cannot know who read it.

Where a secret can leave the pipeline

The secrets store is one control, not four. Each of these routes bypasses masking entirely, and three of them leave a copy that outlives the run.

  • A central box labelled "secrets store, masked" sits at the top, with an arrow down into a box labelled "the CI job".
  • Four arrows leave the job box, each ending in a red box naming one escape route.
  • The first is the build log, reached by an echo or a set -x, and noted as retained and searchable.
  • The second is a traceback, reached when DEBUG is True and settings are rendered on error.
  • The third is an image layer, reached by an ENV instruction or a build argument, and noted as permanent and distributable.
  • The fourth is a pull request from a fork, running with the same secrets, and noted as third-party code.
  • A footer states that masking only hides the exact string, so a base64 of the value or a fragment inside a traceback passes through unmasked.

Which job needs which secret — and which do not

Which job needs which secret — and which do not
JobNeedsMust not have
lint / type checknothingevery credential in the store
unit + integration testsa throwaway CI database passwordproduction database, registry, deploy keys
dependency + secret scannothing (or a scanner token)anything that can deploy
build + push imageregistry credentialsproduction settings and database
deploydeploy credentials, scoped to one environmentthe ability to write to the repository

Together

yaml
deploy:
  environment: production      # protected: approvals + scoped secrets
  permissions: { contents: read, id-token: write }
  # no secrets on the test job at all

Four ways a secret escapes a pipeline that "uses the secrets store"

Four ways a secret escapes a pipeline that "uses the secrets store"
RouteHow it happensWhat stops it
the log`echo $TOKEN`, `set -x`, a debugging line left innever print; masking is a backstop, not a control
a traceback`DEBUG=True` renders settings on errorproduction settings in CI too; `check --deploy`
an image layer`ENV SECRET_KEY=…` or `--build-arg`inject at run time only; `env_file`, never a layer
a forka pull request from outside running with secretsrestricted triggers; no secrets on untrusted runs

Together

bash
# Never do this "just to check":
echo "DATABASE_URL=$DATABASE_URL"
# The log is retained, searchable, and often world-readable.

Remember: The runner holds credentials and runs third-party code, so scope secrets per job and narrow each job's token — the scan and test jobs need no deploy key at all. Assume masking protects only the exact string: never print a secret, keep `DEBUG` off so a traceback cannot render settings, never write one into an image layer or a build argument, and do not let a fork's pull request run with them. Scan the lock file rather than the manifest, before the build, so a vulnerable package never becomes an image. And grade the policy — fail on critical runtime findings, report development-only ones — because a gate that fails on everything is a gate that gets switched off.

See also: the pipeline and where it runs · artifacts approvals and rollback · dependencies secrets and deployment checks · users groups permissions and environment

Advertisement

After the build

Promotion by digest, a human gate, and a rollback that is a command.

Build once, promote, approve — and be able to go back

coreadvanced

Build the image **once**, tag it with the commit, and promote that exact artefact through staging and production — never rebuild per environment. **Artifact management** is keeping those images (and their provenance) long enough to redeploy an old one. **Approvals** are a human gate before a protected environment. A **rollback** is redeploying the previous tag, and it only works when the database schema still accepts the old code.

Think of it as

The rule that everything else hangs off is build once, deploy many. If staging and production are built separately, they are two different artefacts that happen to come from the same commit — different dependency resolutions, different base image digests, different build-time environments — so testing one tells you less than it appears to about the other. Building once and promoting the tag makes the thing you tested and the thing you ship the same object, and it makes the pipeline after the build a series of deployments rather than a series of builds. Tag with something that identifies the source exactly: the commit SHA is the useful primary tag, with a human-friendly release tag pointing at the same digest. Moving tags like `latest` are the opposite of this — they tell you nothing about what is running and they make "redeploy what we had yesterday" unanswerable. Retention follows from that: keep enough images to redeploy any release you might have to go back to, which in practice means keeping tags around considerably longer than you think, because the day you need one is the day you cannot build it. Approvals are the deliberate seam. An environment marked protected requires a named person to approve before the deploy runs, which buys two things: a moment to check the change is the one intended, and an audit record of who released what. Keep it to the environments that need it — an approval on every step turns into a rubber stamp — and make the approval carry the artefact identity, so what is approved is a digest rather than "the pipeline". Rollback is where the honesty lives. Rolling back the *code* is easy and fast: redeploy the previous tag, which is already built and already tested. Rolling back the *database* is not, and the two are not symmetrical. A migration that added a nullable column is harmless to leave in place; a migration that dropped a column, renamed one, or narrowed a constraint has removed something the old code needs, so redeploying the old image fails immediately. That asymmetry is what expand-and-contract exists for — deploy the additive schema change first, then the code, and only remove the old shape in a later release once no running version needs it. The practical rule to carry into every release plan is that the schema must be compatible with both the version you are deploying and the version you would roll back to; when it is not, you no longer have a rollback, you have a forward fix, and the plan should say so out loud before the deploy rather than during the incident.

bash
docker build -t app:$GIT_SHA . && docker push app:$GIT_SHA
# every later stage deploys that tag; nothing rebuilds

What we're doing: Promote one image from staging to production behind an approval, with a rollback that is a command rather than a rebuild.

.github/workflows/release.ymlyaml
name: release
on:
  push: { branches: [main] }

jobs:
  build:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    outputs:
      # The digest, not the tag. Tags can be moved; a digest cannot,
      # so every later job deploys a byte-identical artefact.
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - id: push
        run: |
          docker build -t "ghcr.io/acme/app:${GITHUB_SHA}" .
          docker push "ghcr.io/acme/app:${GITHUB_SHA}"
          echo "digest=$(docker inspect --format='{{index .RepoDigests 0}}' \
            ghcr.io/acme/app:${GITHUB_SHA})" >> "$GITHUB_OUTPUT"

  staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
      # Deploys the digest the build job produced. No build step here —
      # a rebuild would make this a different artefact from the one
      # production is about to run.
      - run: ./deploy.sh --image "${{ needs.build.outputs.digest }}"
      - run: ./smoke.sh --base-url https://staging.acme.test

  production:
    needs: [build, staging]
    runs-on: ubuntu-latest
    # A protected environment: a named reviewer approves before this job
    # starts, and the approval record names the digest above.
    environment: production
    steps:
      # Expand-only migrations, run once, before the new code. The old
      # image must still work against this schema, or there is no
      # rollback and the release plan should have said so.
      - run: ./migrate.sh --image "${{ needs.build.outputs.digest }}"
      - run: ./deploy.sh --image "${{ needs.build.outputs.digest }}"
      - run: ./smoke.sh --base-url https://acme.example

      # Rollback is a redeploy of an image that already exists. It is
      # fast because nothing is built, and possible because retention
      # keeps the previous digests.
      - if: failure()
        run: ./deploy.sh --image "${{ vars.PREVIOUS_DIGEST }}"
9–12
The digest is the identity that cannot move. Passing it between jobs is what makes "the thing we tested" and "the thing we shipped" provably the same object rather than the same tag.
22–30
The staging job deploys and does not build. Any rebuild here would resolve dependencies again and produce a different artefact, which quietly removes the value of having tested it.
33–37
The approval is a property of the environment, so the gate is enforced by the platform rather than by convention — and the record says who approved which digest.
39–43
Migrations run once, ahead of the code, and only expand-shaped changes go out with a release you might roll back. This ordering is what keeps the previous image runnable.
47–51
The rollback path is a deploy of a digest that already exists. It is fast precisely because nothing is rebuilt — and it only works if retention kept the image and the schema still accepts the old code.

Why this works: One artefact is built, tested and promoted by digest; the human gate is enforced by the platform and recorded against that digest; and the rollback is a redeploy that needs no build and no new decision.

Rebuilding the image for each environment

Wrong

yaml
staging:    { steps: [{ run: docker build -t app:staging . }] }
production: { steps: [{ run: docker build -t app:prod . }] }
# two builds, two dependency resolutions, two different artefacts

Better

yaml
build:      { outputs: { digest: ... } }         # built once
staging:    { run: ./deploy.sh --image $DIGEST }
production: { run: ./deploy.sh --image $DIGEST }  # the same digest

What you see: A bug that reproduces in production and not in staging, on the same commit — and a base image or transitive dependency that turns out to have been updated between the two builds.

Why: A build is not a pure function of your repository unless everything it pulls is pinned, and in practice something is not: a base image tag that moved, a package index that published a new version, a system package that updated. Building per environment therefore produces artefacts that can differ in ways no diff of your code will show, which destroys the inference the pipeline exists to support — that a green staging run tells you something about production. Building once makes promotion an operation on an object with a fixed digest, so the only variable left between environments is configuration, which is where you can actually reason about it.

One artefact, three environments, and the way back
CI
registry
staging
approver
production
  1. 1. push app:8f3c2ad (built once)tagged by commit; v1.14.0 points at the same digest
  2. 2. deploy 8f3c2ad
  3. 3. smoke tests passthe artefact is now tested, not the source
  4. 4. production requires approval of digest 8f3c2ad
  5. 5. approved — deploy the same digestno rebuild: promotion, not reconstruction
  6. 6. migrate (expand only)additive, so 8f3c2ac still runs against this schema
  7. 7. health + smoke pass
  8. 8. if not: redeploy app:8f3c2acpossible only because the previous image still exists and the schema is compatible
  1. CI → registry: push app:8f3c2ad (built once) (tagged by commit; v1.14.0 points at the same digest)
  2. registry → staging: deploy 8f3c2ad
  3. staging → CI: smoke tests pass (the artefact is now tested, not the source)
  4. CI → approver: production requires approval of digest 8f3c2ad
  5. approver → production: approved — deploy the same digest (no rebuild: promotion, not reconstruction)
  6. production → production: migrate (expand only) (additive, so 8f3c2ac still runs against this schema)
  7. production → CI: health + smoke pass
  8. CI → production: if not: redeploy app:8f3c2ac (possible only because the previous image still exists and the schema is compatible)

Which schema changes leave you a rollback

Which schema changes leave you a rollback
MigrationOld code still works?Rollback plan
add a nullable columnyes — it ignores the columnredeploy the old tag; leave the column
add a table or an indexyesredeploy; the extra object is harmless
add a NOT NULL column with a defaultyes, if the default is setredeploy; new rows still get the default
**drop** a column the old code reads**no** — every query on it errorsno rollback: forward fix, or restore the column first
**rename** a column**no**expand/contract instead: add, backfill, switch, drop later
narrow a constraint (add UNIQUE)maybe — old code may write duplicatesdrop the constraint before rolling code back

Together

bash
# Reversing a migration is a Django command, but only when the
# operations are reversible — RunPython needs reverse_code to exist.
python manage.py migrate orders 0041

Tagging that answers the questions you will actually ask

Tagging that answers the questions you will actually ask
TagAnswersKeep for
`app:8f3c2ad`exactly which commit is runningas long as it might need redeploying
`app:v1.14.0`which release this is, for humansthe life of the release series
`app:production`nothing — it movesconvenience only; never a rollback target
`app:latest`**nothing at all**do not use it in a deployment

Together

bash
docker build -t app:$GITHUB_SHA .
docker tag app:$GITHUB_SHA app:v1.14.0     # same digest, two names
# deploy references the SHA, always

Remember: Build once, tag by commit, and promote the digest — a rebuild per environment produces a different artefact and quietly invalidates whatever staging told you. Keep images long enough to redeploy any release you might have to go back to, because a rollback that needs a build is not a rollback. Put approvals on protected environments and record which digest was approved. And keep code and schema rollback separate in your head: redeploying the previous image is fast, but it only works if the migration was expand-shaped. When it is not, say so in the release plan — the deployment is forward-fix-only.

See also: the pipeline and where it runs · secrets and dependency scanning · the expand and contract technique · forward reverse and irreversible operations

Advertisement