beginnerintermediateadvanced

AWS · Section 56

AWS for Django / Python Backends

Level: advanced
Read: 35 min
Concepts: 3

A Django or FastAPI project already names everything it needs, so deploying it is mostly a mapping exercise: DATABASES becomes RDS or Aurora PostgreSQL, CACHES becomes ElastiCache, uploads become S3, the Celery broker becomes SQS, anything sensitive becomes Secrets Manager, and stdout becomes CloudWatch Logs. One container image runs both the web service behind an ALB and the worker service with no open ports, which is what stops the two from drifting to different versions, and the task role rather than an access key is the container's identity. Four details then decide whether the deployment is stable rather than merely working. Static files are built with the release, are public and cacheable forever; media is created by users, is private, and is served by presigned URL — treating them as one thing is the most common storage mistake here. Uploads go straight from the browser to S3 so file size never becomes an API constraint. Migrations run as a gated one-off task rather than from the container entrypoint, and because a rolling deployment runs both versions simultaneously, a breaking change has to be split into expand, backfill and contract. Health checks stay cheap and touch no dependency, or one slow query removes every task from the load balancer at once. Finally, the two tiers scale on different signals: the web tier on requests per target or latency, the worker tier on backlog per worker — queue length divided by workers in service, targeted at acceptable latency divided by processing time per message.

AWS overview

AWS Full Course 2026 | AWS Cloud Computing Tutorial for Beginners | AWS Training | SimplilearnSimplilearn

What is true here

Every line of the settings file maps to a service; a line with no service is still running locally.
One image, two commands — web and worker cannot drift to different versions.
Static and media are different in origin, privacy, caching and recoverability.
Migrations are a gated deployment step, expanded before they are contracted.
Web load and job load peak at different times and need separate scaling policies.

What you will be able to do

Map every setting in a Django or FastAPI project to a specific AWS service
Inject configuration and secrets into a task without putting either in the image
Separate static and media correctly, and accept uploads without proxying bytes
Ship a schema change safely through a rolling deployment
Choose and defend a scaling metric for a queue-backed worker tier

From a settings file to a running, separately scaled deployment

Map every setting to a service

DATABASES, CACHES, storage, broker, secrets

Get the deployment details right

static vs media, uploads, migrations, health checks

Scale the tiers separately

requests for web, backlog per worker for jobs

AWS for Django / Python Backends

Mapping a Python backend onto AWS services, the deployment details that differ from running locally, and why the request tier and the job tier need separate scaling policies.

Mapping a Django or FastAPI Backend to AWS

coreadvanced

A Django or FastAPI project already names every piece it needs: a WSGI or ASGI server, a database, a cache, a place for uploads, a task queue, somewhere for settings and secrets. Deploying to AWS is mapping each of those to a managed service and deciding what the process boundaries are. Nothing about the application changes shape — what changes is who runs each piece.

Think of it as

Read your settings file as a shopping list. `DATABASES` becomes RDS, `CACHES` becomes ElastiCache, `DEFAULT_FILE_STORAGE` becomes S3, `CELERY_BROKER_URL` becomes SQS, and everything you read from an environment variable becomes Secrets Manager or Parameter Store. If a line in settings has no service next to it, that line is still running on somebody's laptop.

What we're doing: Turn a working local project into a deployable task definition without changing application code.

mapping.txttext

LOCAL                          AWS
docker-compose up              two ECS services from ONE image

  web:  gunicorn app.wsgi      → service "orders-web"
        ports 8000               command: gunicorn ...
                                 behind ALB target group, port 8000

  worker: celery -A app worker → service "orders-worker"
                                 command: celery -A app worker
                                 no load balancer, no open ports

  db: postgres:16             → RDS PostgreSQL, Multi-AZ
                                 endpoint in the task definition
                                 password in Secrets Manager

  redis: redis:7              → ElastiCache, primary endpoint

  volume ./media              → S3 bucket, private
                                 storage backend swapped in settings

  .env file                   → non-secrets: task definition env
                                 secrets: valueFrom a Secrets ARN

  print() to console          → stdout → awslogs driver → CloudWatch

1: The two-column form is the useful artifact: anything with a blank right-hand column is unfinished work.
9: Same image, different command. This is what makes the worker guaranteed to run the code that was tested.
16: The container gets an endpoint and a secret ARN; it never gets a password in an environment variable.
22: Removing the file log handler is a real code change and the only one on this list — everything else is configuration.

Why this works: Almost every deployment problem at this stage comes from a line in the local setup that has no AWS counterpart — a local directory for uploads, a `.env` committed to the repo, a cron entry on one machine. Writing the mapping out as two columns forces each of those into the open before the first deploy, rather than after the first incident.

Building a separate image for the worker

Wrong

text

# Dockerfile.web     → acme/orders-web:1.4.2
# Dockerfile.worker  → acme/orders-worker:1.4.1   (rebuilt separately)

Better

text

# One image, two task definitions:
#   orders:1.4.2  command ["gunicorn", "app.wsgi", ...]
#   orders:1.4.2  command ["celery", "-A", "app", "worker"]

What you see: A task fails on a model field the web tier already deployed, because the worker image is one build behind and nobody noticed the version skew.

Why: Web and worker share models, migrations and serializers, so they must share a version. Two images means two pipelines that can drift, and the drift is invisible until a worker deserializes a payload the newer web tier produced. One image with two commands makes skew impossible to introduce by accident.

Every line in settings.py, and the service that answers it

Front door and edge

Route 53

api.example.com → ALB alias

CloudFront

static assets and cached GETs

ALB

TLS, health checks, path rules

Your code — one image, two commands

ECS web service

gunicorn / uvicorn, behind the ALB

ECS worker service

celery worker, no load balancer

Scheduled task

celery beat, or an EventBridge schedule

State — none of it in your container

RDS / Aurora PostgreSQL

DATABASES, Multi-AZ

ElastiCache Redis

CACHES, sessions, locks

media uploads and static files

SQS

CELERY_BROKER_URL

Cross-cutting

Secrets Manager

DB password, API keys, SECRET_KEY

IAM task role

the container's only credentials

CloudWatch

logs from stdout, metrics, alarms

The mapping, line by line

The mapping, line by line
In the project	On AWS	What changes in the code
`gunicorn` / `uvicorn` process	ECS task on Fargate (or EC2, or Lambda)	Nothing — it still listens on a port
`DATABASES`	RDS or Aurora PostgreSQL, Multi-AZ	Host becomes an endpoint name; never an IP
`CACHES`	ElastiCache for Redis or Valkey	Host becomes the primary endpoint
`MEDIA_ROOT` uploads	S3 bucket, private, served via presigned URLs	Storage backend swapped; view code unchanged
`STATIC_ROOT` assets	S3 bucket behind CloudFront	Collected at build time, not at runtime
`CELERY_BROKER_URL`	SQS queue (or ElastiCache Redis)	Broker URL only
`SECRET_KEY`, DB password, API keys	Secrets Manager or Parameter Store	Read once at startup by ARN
`logging` handlers	CloudWatch Logs via the container log driver	Log JSON to stdout; delete file handlers
`ALLOWED_HOSTS`, health endpoint	ALB target group health check	Add a cheap `/healthz` that touches nothing

Together

text

# What the container actually receives — endpoints and ARNs, no secrets
DATABASE_HOST      = orders-prod.cluster-abc123.eu-west-1.rds.amazonaws.com
REDIS_HOST         = orders-cache.abc123.ng.0001.euw1.cache.amazonaws.com
MEDIA_BUCKET       = acme-orders-media-prod
CELERY_BROKER_URL  = sqs://
DB_SECRET_ARN      = arn:aws:secretsmanager:eu-west-1:111122223333:secret:orders/db-AbCdEf

# The password itself is fetched at startup using the task role.
# Nothing here is sensitive, so it can live in the task definition.

ALB or API Gateway in front of the same application

ALB or API Gateway in front of the same application
Aspect	ALB → ECS/EC2	API Gateway → Lambda
Process model	Long-running server, many requests per process	One execution environment per concurrent request
Request duration	Bounded by your own timeout settings	Hard ceiling of 900 seconds (15 minutes)
Payload	Large uploads pass through, or bypass via presigned URLs	6 MB request and response, synchronous
Database connections	A pool per process, reused across requests	One per execution environment — reach for RDS Proxy
Idle cost	You pay for running tasks	You pay per request
Reach for it when	Steady traffic, WebSockets, long requests, existing WSGI/ASGI app	Spiky or low traffic, short handlers, no idle spend

Together

text

# The connection-pool difference, concretely
ECS:     8 tasks × 1 pool × 10 connections   =  80 connections, steady
Lambda:  400 concurrent executions × 1 each  = 400 connections, spiky

# The second number is why a Lambda-fronted Django app usually needs
# RDS Proxy, and the first is why an ECS one usually does not.

Remember: Read the settings file as a list of services to rent: RDS for DATABASES, ElastiCache for CACHES, S3 for uploads and static, SQS for the Celery broker, Secrets Manager for anything sensitive, CloudWatch for stdout. One image runs both web and worker; the task role, not an access key, is the container's identity.

Static Files, Media, Jobs, Migrations, and Health Checks

coreadvanced

Once the services are chosen, a handful of details decide whether the deployment is stable. Static files and media files look alike and behave completely differently. Uploads should never travel through your application. Migrations are a deployment step with its own failure mode. Connections to the database are a limited resource. And the health check the load balancer calls decides when your service is considered alive.

Think of it as

Sort everything the application touches into "built once" and "created by users". Static files are built — they ship with the release, are immutable, and can be cached forever. Media is created — it appears at runtime, must survive deploys, and is never public by default. Almost every storage mistake in a Django or FastAPI deployment is these two being treated as one thing.

What we're doing: Deploy a migration that adds a required column, while old and new code are both running.

expand-contract.txttext

The rule: during a rolling deployment, version N and version N+1
serve traffic at the same time. Any migration the old code cannot
survive will break requests, not fail loudly.

RELEASE 1 — expand (safe with old code running)
  Add the column, nullable, with no default backfill:
      ALTER TABLE orders ADD COLUMN channel varchar(20) NULL;
  Old code ignores it. New code writes it when present.
  Migration task runs BEFORE the new tasks start serving.

RELEASE 2 — backfill (no schema change)
  A worker job fills channel for existing rows, in batches,
  with the application still serving normally.

RELEASE 3 — contract (only once nothing reads the old shape)
  ALTER TABLE orders ALTER COLUMN channel SET NOT NULL;
  Now the constraint is safe, because every row has a value and
  every running version writes one.

Doing all three in one release is the common failure: old tasks
insert rows without channel, hit NOT NULL, and return 500s for
the length of the deployment.

1: This is the constraint every rolling-deployment migration answer follows from, and it is easy to forget in staging where only one version runs.
6: Nullable and no default: the cheapest possible change, and the one old code cannot notice.
13: Backfilling from a worker keeps a long-running write off the deployment path entirely.
19: The contract step is the only one that can fail on data, and by now the data is known to be complete.

Why this works: A rolling deployment is defined by both versions being live at once, which turns "the migration ran successfully" into an insufficient test. Expand-then-contract splits a breaking change into three individually safe ones. The cost is three releases; the alternative is a window of 500s on every deploy that touches a constraint.

Running migrations from the container entrypoint

Wrong

text

CMD ["sh", "-c", "python manage.py migrate && gunicorn app.wsgi"]

Better

text

# Pipeline step: run-task with command ["python","manage.py","migrate"]
# Wait for exit code 0, then update the service.
# The service task command is just gunicorn.

What you see: Six tasks start at once, six migrations run concurrently, and the deployment either deadlocks on a lock or half-applies a change while requests are being served.

Why: Every task in the service runs the entrypoint, so the migration runs once per task with no coordination. Making it a one-off task before the service update gives you a single execution, a clear exit code to gate on, and a deployment that stops rather than continuing past a failed migration.

Static files and media files are not the same thing

Static — built with the release

+Produced by `collectstatic` or a frontend build, at build time
+Identical for every user; safe to cache at the edge for a year
+Versioned by content hash, so a deploy never invalidates anything
+Public by design — that is what a CSS file is for
+Lost on rebuild? Rebuild it. Nothing is lost.

Media — created by users at runtime

−Arrives after deploy, from an upload, forever
−Different per user; usually not cacheable and often not shareable
−Keyed by tenant and object id, not by release
−Private by default; access granted per request via a presigned URL
−Lost on rebuild? It is gone. Versioning and backups apply here.

The operational details, and the failure each one prevents

The operational details, and the failure each one prevents
Concern	The choice that works	What it prevents
Static files	Collected at build, uploaded to S3, served via CloudFront	A deploy that serves half-old, half-new assets
Media files	Private S3 bucket, presigned URLs, Block Public Access on	An indexed bucket full of customer documents
Uploads	Presigned PUT direct from the browser	Request timeouts and memory tied to file size
Background jobs	Separate worker service on SQS, own scaling policy	Slow jobs consuming the request-serving capacity
Migrations	One-off task before the new version starts serving	Every task in the deployment running the same migration
Connection pooling	A bounded pool per process; RDS Proxy when processes are many and short-lived	Exhausting `max_connections` during a scale-out
Logging	JSON to stdout, collected by the log driver	Logs living on a filesystem that is deleted with the task
Tracing	Trace ID propagated from the ALB through to the database call	Knowing a request was slow but not which hop was slow
Health checks	A cheap `/healthz` that touches no dependency	A slow database marking every task unhealthy at once

Together

text

# Two health endpoints, two audiences — the distinction that matters
GET /healthz    → 200 if this process can serve. No DB, no cache.
                  Used by the ALB target group.

GET /readyz     → 200 only if DB and cache are reachable.
                  Used by dashboards and alarms, never by the ALB.

# Wire /readyz into the target group and one slow query removes every
# healthy task from the load balancer simultaneously.

Remember: Static is built and cacheable, media is user-created and private — never the same bucket or the same rules. Uploads go direct to S3 by presigned URL. Migrations are a gated one-off task, expand before contract. Health checks stay cheap and dependency-free, and logs go to stdout.

Web and Workers Scale on Different Signals

coreadvanced

The web tier is busy when requests arrive; the worker tier is busy when the queue is deep. Those two things move at different times and for different reasons, so they need two services with two scaling policies. Scaling both on CPU, or running workers inside the web containers, ties them together and makes both worse.

Think of it as

Scale the web tier on how long a user is waiting, and the worker tier on how far behind the queue is. The right worker metric is not queue depth — it is backlog per worker, because the number of workers you need depends on how long each message takes and how much delay is acceptable, not on the raw count.

What we're doing: Explain why one scaling policy for both tiers under-serves both of them.

two-peaks.txttext

A day in one service, both tiers scaled together on CPU:

09:00  Traffic ramps. Requests queue at the ALB, but CPU is
       only 40% because the web tier is waiting on the database.
       No scale-out. p95 latency climbs. Users notice.

11:00  Someone triggers a bulk import. 40,000 messages land on
       the queue. Worker CPU pins, so the shared policy scales
       out — adding web tasks too, which nothing is asking for.

02:00  Nightly reports. The queue is deep for two hours with
       zero request traffic. The shared minimum keeps a full
       web tier running all night to serve nobody, and the
       worker tier scales on a signal the web tier is diluting.

Split into two services with two policies:
  web    → requests per target, min 2, max 20
  worker → backlog per worker (target 100), min 0, max 50

09:00 scales the web tier on the signal that actually moved.
11:00 scales only workers. 02:00 runs 30 workers and 2 web tasks.

1: CPU is the default choice and the wrong one for an I/O-bound web tier — the saturation shows up as latency long before it shows up as CPU.
8: Coupled scaling spends money on the tier that is not busy, every time the other tier is.
15: The overnight case is the clearest: the two tiers have no relationship at all for several hours a day.

Why this works: Web and worker load are driven by different events — one by users arriving, one by work being produced — and they rarely peak together. A single policy has to pick one signal, so it is always reacting late for one tier and over-provisioning the other. Two services cost one extra task definition and remove both problems.

Running the Celery worker inside the web container

Wrong

text

# supervisord in one container: gunicorn AND celery worker
# One ECS service, one scaling policy.

Better

text

# Two ECS services from the same image:
#   orders-web     command: gunicorn ...
#   orders-worker  command: celery -A app worker

What you see: A heavy background job starves the request handlers on the same task, so p95 latency jumps whenever a batch runs — and scaling out to fix it adds more of both processes.

Why: Two processes in one container share CPU, memory and a lifecycle, so they compete during exactly the periods when one of them is busy. They also scale together by definition, which removes the ability to give the queue more capacity without giving the web tier capacity it does not need.

Two services, two signals, one image

The web service reacts to request latency and count; the worker service reacts to how far behind its queue is. Neither signal tells you anything about the other tier.

What each tier scales on, and why the other signal fails

What each tier scales on, and why the other signal fails
Tier	Scale on	Why not the obvious alternative
Web	Requests per target, or p95 latency	CPU lags: a request tier can be saturated on I/O wait at 30% CPU
Worker	Backlog per worker (queue length ÷ workers in service)	Queue depth alone does not scale with worker count, so the target has no stable meaning
Worker, long jobs	Backlog per worker, plus scale-in protection	Plain scale-in can terminate a worker mid-job; the message returns to the queue and the work restarts
Scheduled batch	Scheduled scaling ahead of the known peak	Reactive scaling starts after the backlog exists, so the first minutes are always behind
Both	Separate minimum capacities	A shared minimum sizes one tier for the other tier's idle period

Together

text

# Sizing the worker target, AWS's worked example
acceptable latency        = 10 s
average processing time   =  0.1 s per message
acceptable backlog/worker = 10 / 0.1 = 100 messages   ← the target

# With 1,500 messages visible and 10 workers:
backlog per worker = 1500 / 10 = 150   → above target → scale out

# Note what changes the target: how long a message takes, and how
# long a user will wait. Not the size of the queue.

Remember: Two services, two policies, one image. Scale the web tier on requests per target or latency, and the worker tier on backlog per worker — queue length divided by workers in service — with a target of acceptable latency divided by processing time per message. Protect long-running workers from scale-in.