Filter concepts by levelShowing all levels.

AWS · Python Backends

AWS for Django / Python Backends

Reading a settings file as a shopping list, and the four details that decide whether the deploy is stable

  • One image, two commandsweb and worker can never drift apart
  • Static is built, media is createddifferent buckets, different rules
  • Migrations are a gated taskexpand, backfill, contract
  • Two tiers, two signalsrequests vs backlog per worker

1.Mapping a Django or FastAPI Backend to AWS

A Django or FastAPI project already names every piece it needs: a WSGI or ASGI server, a database, a cache, a place for uploads, a task queue, somewhere for settings and secrets. Deploying to AWS is mapping each of those to a managed service and deciding what the process boundaries are. Nothing about the application changes shape — what changes is who runs each piece.

Key features

  • The application container is identical for web and worker processes; only the command and the scaling policy differ
  • The task role gives the container its AWS permissions — no access keys are ever put in the image or the environment
  • RDS Proxy pools and reuses database connections, which avoids the memory and CPU overhead of opening a new connection per request, and keeps application connections during a failover
  • Secrets are read at startup from Secrets Manager or Parameter Store by ARN, so rotating one is a restart rather than a rebuild
  • ALB fits a long-running Django or FastAPI service; API Gateway fits per-request Lambda handlers, and mixing the two front doors for one service rarely pays off
  • stdout and stderr from the container go to CloudWatch Logs by the log driver, so the application writes JSON to stdout and configures no log files

The mapping, line by line

The mapping, line by line
In the projectOn AWSWhat changes in the code
`gunicorn` / `uvicorn` processECS task on Fargate (or EC2, or Lambda)Nothing — it still listens on a port
`DATABASES`RDS or Aurora PostgreSQL, Multi-AZHost becomes an endpoint name; never an IP
`CACHES`ElastiCache for Redis or ValkeyHost becomes the primary endpoint
`MEDIA_ROOT` uploadsS3 bucket, private, served via presigned URLsStorage backend swapped; view code unchanged
`STATIC_ROOT` assetsS3 bucket behind CloudFrontCollected at build time, not at runtime
`CELERY_BROKER_URL`SQS queue (or ElastiCache Redis)Broker URL only
`SECRET_KEY`, DB password, API keysSecrets Manager or Parameter StoreRead once at startup by ARN
`logging` handlersCloudWatch Logs via the container log driverLog JSON to stdout; delete file handlers
`ALLOWED_HOSTS`, health endpointALB target group health checkAdd a cheap `/healthz` that touches nothing

ALB or API Gateway in front of the same application

ALB or API Gateway in front of the same application
AspectALB → ECS/EC2API Gateway → Lambda
Process modelLong-running server, many requests per processOne execution environment per concurrent request
Request durationBounded by your own timeout settingsHard ceiling of 900 seconds (15 minutes)
PayloadLarge uploads pass through, or bypass via presigned URLs6 MB request and response, synchronous
Database connectionsA pool per process, reused across requestsOne per execution environment — reach for RDS Proxy
Idle costYou pay for running tasksYou pay per request
Reach for it whenSteady traffic, WebSockets, long requests, existing WSGI/ASGI appSpiky or low traffic, short handlers, no idle spend
Every line in settings.py, and the service that answers it

Front door and edge

Route 53

api.example.com → ALB alias

CloudFront

static assets and cached GETs

ALB

TLS, health checks, path rules

Your code — one image, two commands

ECS web service

gunicorn / uvicorn, behind the ALB

ECS worker service

celery worker, no load balancer

Scheduled task

celery beat, or an EventBridge schedule

State — none of it in your container

RDS / Aurora PostgreSQL

DATABASES, Multi-AZ

ElastiCache Redis

CACHES, sessions, locks

S3

media uploads and static files

SQS

CELERY_BROKER_URL

Cross-cutting

Secrets Manager

DB password, API keys, SECRET_KEY

IAM task role

the container's only credentials

CloudWatch

logs from stdout, metrics, alarms

  • Front door and edge
    • Route 53 — api.example.com → ALB alias
    • CloudFront — static assets and cached GETs
    • ALB — TLS, health checks, path rules
  • Your code — one image, two commands
    • ECS web service — gunicorn / uvicorn, behind the ALB
    • ECS worker service — celery worker, no load balancer
    • Scheduled task — celery beat, or an EventBridge schedule
  • State — none of it in your container
    • RDS / Aurora PostgreSQL — DATABASES, Multi-AZ
    • ElastiCache Redis — CACHES, sessions, locks
    • S3 — media uploads and static files
    • SQS — CELERY_BROKER_URL
  • Cross-cutting
    • Secrets Manager — DB password, API keys, SECRET_KEY
    • IAM task role — the container's only credentials
    • CloudWatch — logs from stdout, metrics, alarms

Signature

DATABASES→RDS · CACHES→ElastiCache · media→S3 · broker→SQS · secrets→Secrets Manager · logs→CloudWatch

Read the settings file as a list of services to rent: RDS for DATABASES, ElastiCache for CACHES, S3 for uploads and static, SQS for the Celery broker, Secrets Manager for anything sensitive, CloudWatch for stdout. One image runs both web and worker; the task role, not an access key, is the container's identity.

2.Static Files, Media, Jobs, Migrations, and Health Checks

Once the services are chosen, a handful of details decide whether the deployment is stable. Static files and media files look alike and behave completely differently. Uploads should never travel through your application. Migrations are a deployment step with its own failure mode. Connections to the database are a limited resource. And the health check the load balancer calls decides when your service is considered alive.

Key features

  • Static files are collected at build time into the image or an S3 prefix, versioned per release, and served through CloudFront with a long cache lifetime
  • Media is user-created, lives in a private S3 bucket, and is served by presigned URL or through CloudFront with a signed URL — never by making the bucket public
  • A presigned upload URL lets the browser PUT straight to S3; the URL carries the credentials of whoever signed it and expires with them
  • Migrations run as a one-off task before the new version starts serving, not in the container entrypoint — otherwise every task races every other task
  • Only expand-then-contract migrations are safe during a rolling deployment, because old and new code run at the same time
  • The health check must be cheap and must not touch the database, or one slow query takes every task out of service at once

The operational details, and the failure each one prevents

The operational details, and the failure each one prevents
ConcernThe choice that worksWhat it prevents
Static filesCollected at build, uploaded to S3, served via CloudFrontA deploy that serves half-old, half-new assets
Media filesPrivate S3 bucket, presigned URLs, Block Public Access onAn indexed bucket full of customer documents
UploadsPresigned PUT direct from the browserRequest timeouts and memory tied to file size
Background jobsSeparate worker service on SQS, own scaling policySlow jobs consuming the request-serving capacity
MigrationsOne-off task before the new version starts servingEvery task in the deployment running the same migration
Connection poolingA bounded pool per process; RDS Proxy when processes are many and short-livedExhausting `max_connections` during a scale-out
LoggingJSON to stdout, collected by the log driverLogs living on a filesystem that is deleted with the task
TracingTrace ID propagated from the ALB through to the database callKnowing a request was slow but not which hop was slow
Health checksA cheap `/healthz` that touches no dependencyA slow database marking every task unhealthy at once
Static files and media files are not the same thing

Static — built with the release

  • Produced by `collectstatic` or a frontend build, at build time
  • Identical for every user; safe to cache at the edge for a year
  • Versioned by content hash, so a deploy never invalidates anything
  • Public by design — that is what a CSS file is for
  • Lost on rebuild? Rebuild it. Nothing is lost.

Media — created by users at runtime

  • Arrives after deploy, from an upload, forever
  • Different per user; usually not cacheable and often not shareable
  • Keyed by tenant and object id, not by release
  • Private by default; access granted per request via a presigned URL
  • Lost on rebuild? It is gone. Versioning and backups apply here.
  • Static — built with the release
    • Produced by `collectstatic` or a frontend build, at build time
    • Identical for every user; safe to cache at the edge for a year
    • Versioned by content hash, so a deploy never invalidates anything
    • Public by design — that is what a CSS file is for
    • Lost on rebuild? Rebuild it. Nothing is lost.
  • Media — created by users at runtime
    • Arrives after deploy, from an upload, forever
    • Different per user; usually not cacheable and often not shareable
    • Keyed by tenant and object id, not by release
    • Private by default; access granted per request via a presigned URL
    • Lost on rebuild? It is gone. Versioning and backups apply here.

Signature

static → build-time, public, immutable · media → runtime, private, presigned · migrate → one-off task, expand then contract

Static is built and cacheable, media is user-created and private — never the same bucket or the same rules. Uploads go direct to S3 by presigned URL. Migrations are a gated one-off task, expand before contract. Health checks stay cheap and dependency-free, and logs go to stdout.

3.Web and Workers Scale on Different Signals

The web tier is busy when requests arrive; the worker tier is busy when the queue is deep. Those two things move at different times and for different reasons, so they need two services with two scaling policies. Scaling both on CPU, or running workers inside the web containers, ties them together and makes both worse.

Key features

  • The number of messages in a queue does not change proportionally to the number of workers, which is why queue depth alone is a poor target-tracking metric
  • Backlog per instance is the queue length divided by the number of instances in service; that is the metric to target
  • Acceptable backlog per instance is acceptable latency divided by average processing time per message — AWS's worked example is 10 s ÷ 0.1 s = 100 messages
  • Web traffic and job volume peak at different times: a nightly report run creates worker load with no request load at all
  • Workers handling long tasks can use scale-in protection so a scale-in event does not kill a job halfway through
  • A message not finished when a worker is terminated returns to the queue and is picked up by another worker

What each tier scales on, and why the other signal fails

What each tier scales on, and why the other signal fails
TierScale onWhy not the obvious alternative
WebRequests per target, or p95 latencyCPU lags: a request tier can be saturated on I/O wait at 30% CPU
WorkerBacklog per worker (queue length ÷ workers in service)Queue depth alone does not scale with worker count, so the target has no stable meaning
Worker, long jobsBacklog per worker, plus scale-in protectionPlain scale-in can terminate a worker mid-job; the message returns to the queue and the work restarts
Scheduled batchScheduled scaling ahead of the known peakReactive scaling starts after the backlog exists, so the first minutes are always behind
BothSeparate minimum capacitiesA shared minimum sizes one tier for the other tier's idle period
Two services, two signals, one image

The web service reacts to request latency and count; the worker service reacts to how far behind its queue is. Neither signal tells you anything about the other tier.

  • Two independent scaling loops drawn side by side.
  • On the left, the web service: users send requests through an ALB to web tasks. Its CloudWatch signal is request count per target and p95 latency, which drives a target-tracking policy that adjusts the number of web tasks.
  • On the right, the worker service: producers put messages on an SQS queue, and worker tasks drain it. Its signal is backlog per worker — queue length divided by workers in service — which drives its own target-tracking policy.
  • Both services run the same container image, shown at the bottom.
  • Note: a nightly batch fills the queue with no web traffic at all, and a marketing spike fills the web tier with no queue depth at all.

Signature

web → requests per target / p95 latency · worker → backlog per worker (queue length ÷ workers in service)

Two services, two policies, one image. Scale the web tier on requests per target or latency, and the worker tier on backlog per worker — queue length divided by workers in service — with a target of acceptable latency divided by processing time per message. Protect long-running workers from scale-in.

Remember

  • The task role is the container's identity — no access keys in the image, and secrets injected by ARN rather than as plain environment values.
  • RDS Proxy exists because pooling avoids the memory and CPU overhead of a new connection per request, and it keeps connections through a failover.
  • A rolling deployment runs both versions at once, so only expand-then-contract migrations are safe.
  • Acceptable backlog per worker is acceptable latency divided by processing time per message — AWS's example is 10 s ÷ 0.1 s = 100.

Pro tip

  • Write the local-to-AWS mapping as two columns before touching any infrastructure: every row with an empty right-hand side is a deployment surprise you have not had yet.
  • Keep two health endpoints — a dependency-free one for the load balancer, and a dependency-checking one for alarms. Wiring the second into the target group turns a database blip into an outage.

Next up

  • AWS for Full-Stack / React Applications
  • ECS Deployment Architecture
  • Production Debugging on AWS

Sources

  • Amazon RDS Proxy — docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
  • Scaling policy based on Amazon SQS — docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html
  • Download and upload objects with presigned URLs — docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
  • Checked against official AWS documentation 2026-09-04

Spotted a mistake on this page? Tell us.