Deployment Strategies cheat sheet · Django

1.Rolling, blue/green and canary

A **rolling** deploy replaces instances a few at a time, so both versions serve traffic while it runs. **Blue/green** brings up a whole second fleet, tests it, then switches all traffic at once — and switches back just as fast. A **canary** sends a small share of real traffic to the new version, watches the error rate, and only then continues. All three run two versions of your code against **one** database.

Key features

All three run two code versions against one database — the schema must fit both
Rolling: no extra capacity, both versions live, rollback is another full roll
Blue/green: double capacity briefly, instant switch, instant switch back
Canary: real evidence before full exposure — but only with per-version metrics
Sessions, cache values and queued task payloads must be readable by both versions
Workers are a third fleet: old consumers will read messages the new code wrote

The three strategies, compared on what they actually cost

The three strategies, compared on what they actually cost
Dimension	Rolling	Blue/green	Canary
extra capacity	none	2× during the switch	a little
both versions live	yes, throughout	briefly, at the switch	yes, by design
blast radius of a bad release	grows as it rolls	everyone, instantly	the canary share only
rollback	another full roll	flip back — seconds	stop and drain the canary
evidence before full exposure	none	smoke tests, no real traffic	real traffic, real users
needs	a health check	a second fleet and a switch	traffic splitting + per-version metrics

What must be compatible across the two live versions

What must be compatible across the two live versions
Shared thing	The failure if it is not	The discipline
database schema	old code queries a column the new release dropped	expand → migrate → switch → contract, one release later
session contents	random logouts, or a `KeyError` on a session key	add keys; never repurpose or remove one in the same release
cache values	a pickled object the other version cannot read	version the cache key when the shape changes
queued task payloads	old worker gets an argument it has no parameter for	add optional arguments only; never rename a task in place
static asset URLs	a page from v2 requests an asset only v2 collected	hashed filenames, and keep the previous build served

Share of traffic on the new version, over the deploy

One variable separates the three: how fast the new version reaches 100% of traffic. Everything else — capacity cost, blast radius, speed of rollback — follows from that curve.

Command

bash

./deploy.sh --strategy canary --share 5 --watch-minutes 15

The three strategies differ only in how fast the new version reaches 100% of traffic: rolling ramps with no extra capacity, blue/green steps with double capacity and the fastest rollback, and a canary holds a small share while you gather real evidence. All of them run two versions of the code against one database, one cache and one queue, so the schema, the session format, the cache values and the task payloads must all satisfy both versions at once — which is expand-and-contract, not a strategy choice. And a canary only works with per-version metrics; without them, five per cent of the errors disappears into normal variation.

2.Feature flags: deploying is not releasing

A **feature flag** is a runtime switch around new behaviour. The code ships to production turned off, and you turn it on separately — for internal users first, then a percentage, then everyone. That splits one risky event into two: **deploying** the code, which is now boring, and **releasing** the behaviour, which is reversible in seconds without a deploy.

Key features

Deploy the code dark, then release the behaviour — two events, two risks
Turning a flag off takes effect at the next request; a rollback takes a pipeline run
Four kinds: release (temporary), operational (permanent), experiment, permission
A "permission flag" is authorisation — it belongs in the permission model, not here
Every flag doubles the state space, so it needs an owner and a removal date
Evaluate from cache, and define the value for when the flag store is unreachable

Four things people call a feature flag

Four things people call a feature flag
Kind	Lifetime	Owned by	Belongs in
release flag	weeks — delete after 100%	the team shipping the change	a flag store
operational flag	permanent	whoever is on call	a flag store, with a runbook entry
experiment flag	the length of the experiment	whoever reads the results	an experiment tool
permission flag	the life of the account	the product	your permission model, not a flag store

One event, or two

Deploy = release

New code and new behaviour arrive in the same minute
A problem could be the build, the config or the feature — you cannot tell
Undo is a pipeline run: minutes, and only if the schema allows it
Every change waits for a deploy window, so changes get batched
Batched changes make the next incident harder to attribute

Deploy, then release

Code ships dark; the deploy proves only that nothing broke
Behaviour is enabled for staff, then 5%, then everyone
Undo is a flag write: effective on the next request
A bad feature does not force a rollback of unrelated changes
The cost is a branch in the code, which you must later delete

Command

python

if flags.enabled("checkout.new-tax-engine", user=request.user):

A flag splits one risky event into two: the deploy, which becomes boring because the code ships dark, and the release, which you control and can reverse at the next request instead of at the next pipeline run. Know which kind you are adding — a release flag is temporary and must be deleted, an operational flag is a permanent degradation switch, and a "permission flag" is authorisation that belongs in your permission model. Evaluate from cache so a flag in a loop is not an N+1, decide what the value is when the store is unreachable, and bucket users by a stable hash so a percentage rollout does not flicker.

3.Draining, health checks and the smoke test that decides

A **graceful shutdown** stops accepting new requests, finishes the ones in flight, and only then exits. The **health check** is what the load balancer polls to decide whether to send you traffic — so failing it *first*, while still serving, is what makes the drain invisible. A **smoke test** is the handful of real requests run against the new deployment to decide whether the release stands or is rolled back.

Key features

Fail readiness first, keep serving, then stop accepting — that order is the drain
Wait at least one balancer check interval before shutting down, or you race it
Grace periods nest: gunicorn graceful-timeout < systemd/platform grace period
Readiness may check the database; liveness must not, or a blip restarts the fleet
Smoke tests are the first evidence the artefact works *in this environment*
A smoke failure that does not trigger a rollback is a notification, not a gate

The shutdown sequence, and what breaks if you skip a step

The shutdown sequence, and what breaks if you skip a step
Step	What it does	Skip it and
fail readiness	tells the balancer to stop routing here	new requests arrive at a draining process
keep serving	covers the balancer's next check interval	you race the balancer and lose
SIGTERM to the master	begins gunicorn's graceful shutdown	—
finish in-flight requests	up to `--graceful-timeout`	responses are cut off mid-write
exit	the process ends before the platform's deadline	the platform sends SIGKILL instead

Three checks that get confused

Three checks that get confused
Check	Asks	Failure means
liveness	is this process wedged?	restart the container
readiness	should this instance get traffic?	remove it from the pool
smoke test	does this release actually work here?	roll the release back

Replacing one instance without dropping a request

t+0s
Drain flag set; readiness starts failing
the process is still serving normally — this is an announcement, not a state change
t+0s → t+10s
The balancer notices
one check interval, plus a margin. New requests stop arriving; in-flight ones continue
t+10s
SIGTERM to the gunicorn master
gunicorn: "graceful shutdown; waits for workers to finish requests up to graceful_timeout"
t+10s → t+40s
Draining
no new connections are accepted; typically finishes in a second or two because nothing new is arriving
t+40s
Force kill of anything still running
gunicorn kills stragglers at the graceful timeout — the window is a guarantee, not a plan
t+55s
Platform grace period would fire
deliberately later than every step above; if it fired first, it would kill a healthy drain
after
New instance starts, readiness passes, smoke test runs
the balancer adds it only once readiness passes; the smoke result decides whether the release stands

Command

bash

POST /internal/drain   ->   readiness 503, still serving   ->   SIGTERM

Announce unavailability before becoming unavailable: fail readiness, keep serving for at least one balancer check interval, then `SIGTERM`, then drain, then exit — and make sure the platform's grace period is longer than the sum, or it sends the kill instead. Readiness may check dependencies and should be cached; liveness must not touch the database, or a blip restarts the fleet. Then run a smoke test against the deployed environment, because that is the first evidence this artefact works with real configuration, real credentials and real static files — assert on content rather than status, and wire the failure to the rollback, or it is a notification rather than a gate.

4.Rollback planning, and why migrations make it hard

A rollback plan is a sentence written **before** the deploy: which version we go back to, what makes us decide, and whether going back actually works. It often does not, because a deploy changes two things with different reversibility — the code, which is a redeploy away, and the database, which is not. Some changes have no rollback at all, and the time to discover that is while planning, not during an incident.

Key features

A rollback plan names the version, the trigger and whether going back works
Additive schema → rollback is a redeploy. Destructive schema → forward fix only
Reversing a migration restores the shape, not the values — a data transform is one-way
Emitted side effects (email, payment, webhook) need compensation, not rollback
Write the point at which rollback stops being possible into the release notes
A feature flag is the reversible alternative when the schema cannot be

Four release shapes, and what "roll back" means for each

Four release shapes, and what "roll back" means for each
Release contains	Rollback	Plan
code only	redeploy the previous digest	the ordinary case — seconds, no schema involved
additive migration	redeploy; leave the new column	previous version never mentions it, so it is inert
destructive migration	none	forward fix only — say so before deploying
data transform / deletion	none (shape can return, values cannot)	take a verified backup; plan a restore path
side effects emitted	compensating action, not a rollback	a refund, a correction, a tombstone event

Making an unrollbackable change rollbackable

Making an unrollbackable change rollbackable
Instead of	Do	Because
drop a column in the release	drop it two releases later	the previous version stops needing it first
rename a column	add, dual-write, switch reads, drop later	a rename is a drop and an add at the same instant
add `NOT NULL` to an existing column	add nullable, backfill, then constrain	the constraint fails on rows the old code still writes
change behaviour in the deploy	ship dark behind a flag	a flag is reversible at the next request
rewrite rows in a migration	backfill in a resumable command	a migration that fails halfway leaves neither shape

What kind of undo a release actually has

side effects already emittedno side effects emitted

schema: additive onlyschema: destructive

Redeploy the previous digest: The ordinary release. The old image still runs against this schema, so rollback is one command and takes seconds.
Roll back, then compensate: Code goes back cleanly, but 400 emails were sent and 12 payments captured. The undo is a product decision: refunds, corrections, a tombstone event.
Forward fix only: A dropped or renamed column means the previous version cannot start. Say this in the release notes before deploying, not during the incident.
Restore, not rollback: Destroyed data plus emitted effects. This needs a verified backup, a restore rehearsal and a communications plan — it is an incident procedure, not a deploy step.

Command

text

Rollback: redeploy <digest> · Trigger: <observation> · Valid until: <release>

Write the rollback plan with the change, not after it: the version and digest, the command, the trigger that decides, and — most importantly — what going back will *not* fix. A deploy changes two things with different reversibility, so classify it: additive schema means a redeploy is enough, destructive schema means forward fix only, a data transform cannot be undone by reversing a migration because the shape returns without the values, and emitted side effects need compensating actions rather than a rollback. Name the release where the plan expires. And when a change has no rollback, use the two tools that give you one: a feature flag, and expand-and-contract.