Filter concepts by levelShowing all levels.

Django · Section 91

Linux for Django Developers

Level
intermediate
Read
40 min
Concepts
5

Everything in this section exists because a Django deployment is a tree of ordinary Linux processes, and most production questions are questions about that tree. systemd starts one gunicorn master; the master forks N workers; each worker imports your settings and opens its own database connections. "Own" is the load-bearing word: a module-level cache is per worker, `CONN_MAX_AGE` keeps one connection per worker, and a memory leak grows in one process and is reclaimed only when that process exits. PIDs address the nodes of that tree, and they are kernel-assigned and reused, so a PID is fine to type now and never safe to store. Signals are the interface: `SIGTERM` asks gunicorn to drain up to its graceful timeout, `SIGHUP` reloads configuration and replaces workers without closing the listening socket, and `SIGKILL` — which signal(7) says "cannot be caught, blocked, or ignored" — loses every request in flight because no cleanup code runs at all. Threads share memory and, under the GIL, buy concurrency while waiting on I/O rather than parallel CPU, which is why "add threads" and "add workers" solve different problems. Permissions decide what that tree may touch, and they are checked owner, then group, then other, with the first match winning. Run the app as a dedicated `nologin` system user, own the tree at `750`, keep the secrets file at `600`, and never make a media directory both writable and executable. Remember that on a directory `x` means "may traverse", which is why a recursive `644` breaks static serving. Configuration arrives through the environment so one image runs in every environment — never as an `ENV` layer holding a secret, because an image layer is permanent and distributable. When something is wrong, the tools split by question. `ps` is a snapshot for structure; `top` and `htop` are the same data re-read for consumption; `lsof` answers "who holds this port or file"; `kill` sends a signal rather than killing. Read RSS, which ps defines as "the non-swapped physical memory that a task has used", and ignore VSZ, which is address space and is routinely ten times larger. `grep` selects lines, `awk` selects fields, `sed` transforms — and on JSON logs, parse rather than pattern-match, because `grep` will happily match inside a value. `curl -i` shows what the server actually returned and `-H` sends the header a proxy would have added; `ssh host "cmd"` runs one command and hands back its exit status, which is what makes it scriptable. Finally, two things start work without you. systemd handles boot, crash recovery (`Restart=always`, `RestartSec=`) and reload, and its `TimeoutStopSec` must exceed your graceful timeout or systemd is the thing killing draining workers. cron runs with almost no environment — no virtualenv, barely any `PATH`, the home directory as its working directory — so the line that works pasted into a shell fails at 03:00 unless every path is absolute, the settings module is explicit, and the output goes somewhere you read.

What is true here

  1. The deployment is a process tree; memory and connections are per process.
  2. Signal the master: TERM drains, HUP reloads, KILL cannot be caught.
  3. The user the app runs as is the whole of its file access — and its blast radius.
  4. RSS is memory, VSZ is address space; comparing RSS across equal-age workers finds leaks.
  5. systemd owns restart and reload; cron owns nothing about your environment.

What you will be able to do

  • Read a gunicorn/Celery process tree and explain its memory and connection count
  • Reload a running server without dropping a connection
  • Lay out users, groups and modes so the app can read its secrets and nothing else can
  • Triage a slow box from the machine inward, using the right tool per question
  • Write a unit file and a cron line that still work when you are asleep
From boot to a served request — and the command that addresses each hop
ExecStartfork ×NUser= /Group=may open only whatthat user maystdout/stderra second, emptierenvironmentthe safedeploythe unsafestopnew PIDs, nodropped connection

Boot

systemd starts every unit marked `enable`d

gunicorn.service

User=deploy · EnvironmentFile=/srv/app/.env · Restart=always

gunicorn master

owns the listening socket — the only thing you signal

N worker processes

own memory, own DB connections, own leaked bytes

File access

decided by the user in `User=`, not by the code

The journal

journalctl -u gunicorn -f · --since · -p err

cron / systemd timer

starts work with none of your shell environment

systemctl reload

ExecReload → SIGHUP → workers replaced, socket kept

kill -9

cannot be caught: in-flight requests are simply lost

  • Boot — systemd starts every unit marked `enable`d
    • leads to gunicorn.service
  • gunicorn.service — User=deploy · EnvironmentFile=/srv/app/.env · Restart=always
    • leads to gunicorn master (ExecStart)
    • leads to File access (User= / Group=)
    • leads to cron / systemd timer (a second, emptier environment)
  • gunicorn master — owns the listening socket — the only thing you signal
    • leads to N worker processes (fork ×N)
    • leads to systemctl reload (the safe deploy)
  • N worker processes — own memory, own DB connections, own leaked bytes
    • leads to File access (may open only what that user may)
    • leads to The journal (stdout/stderr)
    • on error, leads to kill -9 (the unsafe stop)
  • File access — decided by the user in `User=`, not by the code
  • The journal — journalctl -u gunicorn -f · --since · -p err
  • cron / systemd timer — starts work with none of your shell environment
  • systemctl reload — ExecReload → SIGHUP → workers replaced, socket kept
    • leads to N worker processes (new PIDs, no dropped connection)
  • kill -9 — cannot be caught: in-flight requests are simply lost

The tree, and who may touch what

Processes, PIDs, signals and threads — then the user, group and mode that bound them.

Processes, PIDs, signals and threads

coreintermediate

A **process** is one running program with its own memory. Its **PID** is the number the kernel gives it, and the number every tool asks you for. A **signal** is a one-byte message you send to a process — `SIGTERM` asks it to stop, `SIGKILL` makes the kernel stop it. A **thread** runs inside a process and shares that process's memory, which is why threads are cheaper and why one thread's crash can take the whole process with it.

Think of it as

A Django deployment is a small tree of processes, and almost every operational question is really a question about that tree. systemd starts one gunicorn *master*; the master forks N *workers*; each worker imports your settings, opens its own database connections and serves requests. The consequences follow directly from "own memory". A module-level cache is per worker, so eight workers hold eight copies and a value written in one is invisible in the others. `CONN_MAX_AGE` keeps a connection per worker, so connection count is a function of process count, not of traffic. A memory leak grows in one worker and is fixed by that worker exiting, which is what `--max-requests` automates. PIDs are how you address one node of that tree. They are assigned by the kernel, reused after a process exits, and they change on every restart — so a PID is fine to type into `kill` right now and useless to store in a config file. The parent PID (PPID) is what lets you read the tree: every worker's PPID is the master's PID, which is how `ps -f` shows you the indentation. Signals are the interface to a running process, and only a few matter. `SIGTERM` (15) is the polite stop: gunicorn treats it as graceful shutdown, waiting for workers to finish in-flight requests up to `graceful_timeout`. `SIGKILL` (9) cannot be caught, blocked or ignored — the process does not run another instruction, so no cleanup happens, no connection is closed politely, and any in-flight request is simply lost. That asymmetry is the reason a deploy sends TERM first and KILL only as a last resort, and the reason your container platform's "termination grace period" must be longer than your graceful timeout, or the platform is the one sending the KILL. `SIGHUP` (1) is conventionally "reload your configuration", and gunicorn implements exactly that. Threads share the process's memory, so they are cheap to create and communicate through ordinary variables — but in CPython the global interpreter lock means threads do not give you parallel CPU work; they give you concurrency while waiting on I/O. That is why gunicorn's `gthread` worker helps a database-bound Django app and does nothing for a CPU-bound one, and why "add threads" and "add workers" are answers to different problems.

bash
kill -TERM <pid>     # ask; the process may clean up
kill -KILL <pid>     # the kernel stops it; nothing runs, nothing is saved

What we're doing: Find the gunicorn master on a box you have just been handed, read its worker tree, and reload it without dropping a request.

a five-minute incident sessionbash
# PIDs and sizes below are illustrative; the columns are what ps -o prints.
# 1. Which process is serving the site, and what is its PID?
$ pgrep -a -f "gunicorn: master"
4812 gunicorn: master [config.wsgi]

# 2. Read the tree. Every worker's PPID is the master's PID — that
#    parent/child link is what makes one signal able to drain them all.
$ ps -o pid,ppid,rss,etime,args -C gunicorn
  PID  PPID   RSS     ELAPSED COMMAND
 4812     1 62104    03:11:47 gunicorn: master [config.wsgi]
 4813  4812 189320   03:11:46 gunicorn: worker [config.wsgi]
 4814  4812 402884   03:11:46 gunicorn: worker [config.wsgi]
 4815  4812 191002   03:11:46 gunicorn: worker [config.wsgi]

# 3. Worker 4814 is holding ~400 MB against its siblings' ~190 MB.
#    That is per-process memory: a leak in one worker is invisible in
#    the others, and exiting that worker is what reclaims it.

# 4. Reload configuration and code with no dropped connection. The
#    master keeps the listening socket, starts new workers, and
#    gracefully stops the old ones.
$ kill -HUP 4812

# 5. Watch the replacement happen. New PIDs, elapsed time back to zero,
#    and 4814's memory gone with it.
$ ps -o pid,ppid,rss,etime --ppid 4812
  PID  PPID   RSS     ELAPSED
 4901  4812 121440       00:04
 4902  4812 119880       00:04
 4903  4812 120106       00:04
3–4
`pgrep -a -f` matches against the full command line, which is how you find a process by what it *is* rather than by a PID you would have to have known already.
8–13
The PPID column is the tree. Every worker points at 4812, so signalling 4812 is signalling the group — you almost never signal an individual worker.
15–17
RSS is resident memory per process — `ps` documents it as "the non-swapped physical memory that a task has used". Three workers running the same code with very different RSS is the signature of a per-request leak, and it is why `--max-requests` recycles workers.
19–22
HUP is gunicorn's documented reload: "reload configuration, spawn new workers, and gracefully stop old ones". The listening socket never closes, so no client sees a refused connection.
24–30
New PIDs prove the workers were replaced rather than restarted in place, and the reset RSS proves the leaked memory went with the old process.

Why this works: You located the running server without prior knowledge, read the parent/child structure that explains per-worker memory and connections, and replaced every worker without dropping a connection.

Killing the workers instead of the master

Wrong

bash
$ pkill -9 -f "gunicorn: worker"
# the master immediately respawns them — and every in-flight request is lost

Better

bash
$ kill -HUP 4812     # replace workers gracefully
$ kill -TERM 4812    # or drain and stop the whole server

What you see: A burst of 502s in the load balancer log at the exact second you ran the command, and application logs that simply stop mid-request with no traceback.

Why: The master exists to own the listening socket and supervise its children, so it is the correct address for every lifecycle instruction. Killing workers directly bypasses the drain: `-9` is `SIGKILL`, which "cannot be caught, blocked, or ignored", so the worker does not finish the response it is writing, does not close its database connection, and does not run any shutdown hook. The master then does exactly its job and forks replacements, which makes the damage look self-healing while every request in flight was dropped. Signal the master and let it manage its own children.

The process tree a Django deployment actually is

Every worker is a separate process with its own memory and its own database connections. Signals go to the master, which is what makes a graceful restart possible at all.

  • A tree diagram. At the top, systemd, PID 1, is the parent of the gunicorn master process, PID 4812.
  • The gunicorn master has three children drawn below it: worker processes with PIDs 4813, 4814 and 4815. Each worker box notes that it holds its own memory and its own database connection.
  • A separate branch on the right shows a Celery master, PID 5001, with one worker child, PID 5002, drawn to show that background work is its own process tree.
  • An arrow labelled SIGTERM points at the gunicorn master only, with a note that the master forwards a graceful stop to its workers.
  • A red note states that SIGKILL sent to a worker loses that worker in-flight requests, because no cleanup code runs.

The signals worth knowing, and what gunicorn does with each

The signals worth knowing, and what gunicorn does with each
SignalNo.Default actionWhat gunicorn's master does
`SIGTERM`15Termgraceful shutdown, up to `graceful_timeout`
`SIGINT` / `SIGQUIT`2 / 3Term / Corequick shutdown
`SIGHUP`1Termreload config, spawn new workers, stop old ones
`SIGTTIN` / `SIGTTOU`21 / 22Stopincrease / decrease the worker count by one
`SIGUSR1`10Termreopen log files (what logrotate sends)
`SIGKILL`9Term**nothing** — it cannot be caught; the kernel stops it

Together

bash
kill -TERM 4812        # ask gunicorn's master to drain and stop
kill -HUP 4812         # reload config without dropping the listening socket
kill -TTIN 4812        # one more worker, right now

Process or thread — what you actually get

Process or thread — what you actually get
DimensionExtra process (worker)Extra thread
memorya full copy — RSS × Nshared with its process
a crash takes downthat worker only; the master respawns itusually the whole process
DB connections**one set per process**one per thread, from the same process
CPU-bound workgenuinely parallelserialised by the GIL
I/O-bound workparallelconcurrent — this is the case threads help

Together

bash
gunicorn config.wsgi -w 4 -k gthread --threads 2
# 4 processes x 2 threads = up to 8 concurrent requests, 4 memory copies

Remember: A Django deployment is a process tree: systemd → gunicorn master → N workers, each with its own memory and its own database connections — which is why connection count follows process count, not traffic. PIDs are kernel-assigned and reused, so never store one. Signal the master, not the workers: `SIGTERM` drains, `SIGHUP` reloads, and `SIGKILL` (9) "cannot be caught, blocked, or ignored", so it loses every in-flight request. Threads share memory and, under the GIL, buy concurrency while waiting on I/O — not parallel CPU.

See also: looking at a running box · systemd the journal and cron · worker multiplication and connection exhaustion · servers workers and lifecycle

Users, groups, permissions and the environment

coreintermediate

Every file has an **owner**, a **group**, and three sets of permissions — read, write, execute — for the owner, the group, and everyone else. Every process runs *as* a user, and that is what decides which files it may touch. **Environment variables** are name/value pairs the process inherits when it starts; they are where a Django deployment keeps its secrets, because they are not in the repository.

Think of it as

Permissions answer one question: this process runs as this user, may it open this file? The answer is computed from three fields you can read directly in `ls -l` — the mode string, the owner, and the group. If the process's user is the owner, the owner bits apply and the others are ignored; otherwise, if the user is in the file's group, the group bits apply; otherwise the "other" bits apply. That order is why adding permissions for "everyone" does not help a user who is already the owner with no bits set. The numeric form is the same information: read is 4, write is 2, execute is 1, and you add them per column, so `640` is owner read+write, group read, no access for anyone else. On a directory the meanings shift in a way worth memorising once — read lists the names inside, write creates and deletes entries, and execute is what allows *traversing* into it. A directory with read but not execute lets you see filenames and open none of them, which is exactly the shape of the "permission denied on a file that clearly exists" bug. In a Django deployment this all lands on three or four concrete decisions. Run gunicorn as a dedicated unprivileged user, never as root, so a remote-code-execution bug in a dependency inherits that user's limited reach. Give the media upload directory write permission for that user and nothing else, because it is the one directory where untrusted bytes land. Give log files a group the log shipper is in, so it can read without being able to write. And put the secrets in a file owned by the deploy user with mode `600`, which is the difference between "a secret" and "a secret every account on the box can read". Environment variables are the delivery mechanism because they keep configuration out of the image and out of git — the same image runs in staging and production, differing only in what it is handed at start-up. They are not encrypted and they are not private: anything that can read `/proc/<pid>/environ` as that user, and anything that dumps the environment into an error page or a log line, has the secret. So the rule is that the environment is a fine place to *pass* a secret and a bad place to *print* one.

bash
chown deploy:webapp /srv/app/.env && chmod 600 /srv/app/.env

What we're doing: Lay out a Django box so the app runs unprivileged, the secrets are unreadable to anyone else, and nginx can still serve static files.

provision.shbash
# 1. A dedicated user with no login shell and no home directory to
#    write into. A remote-code-execution bug inherits THIS user, so the
#    less it can do, the smaller the incident.
useradd --system --no-create-home --shell /usr/sbin/nologin deploy
usermod -aG webapp deploy          # a group nginx is also in

# 2. The application tree: the deploy user owns it, the group may enter
#    and read, nobody else gets in at all.
chown -R deploy:webapp /srv/app
chmod 750 /srv/app

# 3. The secrets file. 600 means owner read/write and nothing else —
#    not group, not other, not any other account on this machine.
install -o deploy -g webapp -m 600 /dev/null /srv/app/.env
printf 'DJANGO_SECRET_KEY=%s\n' "$GENERATED_KEY" >> /srv/app/.env

# 4. Media: the ONE directory the app writes to. Writable by the app
#    user, and never executable — an uploaded .py under 777 in a path
#    the server can execute is a remote shell.
chown deploy:webapp /srv/app/media
chmod 750 /srv/app/media

# 5. Static files are read by nginx, which runs as a different user, so
#    the directories need traverse (x) and the files need read.
chmod 755 /srv/app/static
find /srv/app/static -type f -exec chmod 644 {} +

# 6. Confirm from the app's point of view, not from root's. root can
#    read everything, so testing as root proves nothing.
sudo -u deploy test -r /srv/app/.env && echo "app can read its secrets"
sudo -u www-data test -r /srv/app/.env || echo "nginx cannot — correct"
1–4
`--system` with `nologin` creates an account that runs services and cannot be logged into. This is the single highest-value line in the file: everything an attacker gains through the application is bounded by what this user may do.
9–10
`750` on the directory: the owner may enter, list and write; the group may enter and list; everyone else is refused at the directory, so the files inside never come into play.
13–14
`install -m 600` creates the file with the right mode from the start. Creating it first and `chmod`-ing after leaves a window — usually milliseconds, occasionally a whole deploy — where the secret is world-readable.
17–21
Media is the only path where untrusted bytes are written to disk, which is exactly why it must not be executable and must not be `777`. The upload validation concept covers the other half of this.
25–26
Directories need `x` for a *different* user to traverse them; files only need `r`. This split is why `chmod -R 644` on a static tree breaks nginx — it strips traverse from the directories.
30–31
Verify as the user that will actually run, using `sudo -u`. Every permission test done as root passes, which makes root the worst possible account to test from.

Why this works: The application runs as an account that can read its own secrets and write exactly one directory; nginx can serve static files and cannot read the secrets; and both facts were checked as the users involved rather than as root.

chmod 777 to make the upload error go away

Wrong

bash
chmod -R 777 /srv/app/media    # "now everyone can write, problem solved"

Better

bash
chown -R deploy:webapp /srv/app/media && chmod -R 750 /srv/app/media

What you see: Uploads start working, and months later a file that was uploaded through the media path is executed by the web server, or edited by an unrelated account on the same host.

Why: `777` grants write to every account on the machine, including any service that has been compromised and any user who should only be reading. The original error — permission denied on write — was telling you that the directory belonged to the wrong user, and the fix is to change the owner rather than to remove the check. The executable bit matters independently: media holds bytes supplied by strangers, so a directory that is both writable and executable is a place where an attacker can put a file and then get it run. Own it as the app user, mode `750`, and never set `x` on the files.

One line of `ls -l`, field by field

-rw------- 1 deploy webapp 412 Sep 5 09:41 /srv/app/.env

-rw-------

Type and mode — Leading `-` is a regular file (`d` is a directory). Then three triples: owner `rw-`, group `---`, other `---`. Numerically `600`. Nobody but the owner can read this file, which is the whole point of it.

deploy

Owner — The user the file belongs to. gunicorn runs as this user, so the owner bits are the ones that apply to it — and to nothing else on the box.

webapp

Group — The group bits here are `---`, so group membership grants nothing. Set group read (`640`) only when another service — a log shipper, nginx — genuinely needs it.

412

Size in bytes — Small, because it holds settings and not data. A `.env` that has grown to kilobytes usually means application data has leaked into configuration.

/srv/app/.env

Path — Outside the repository checkout and outside the container image. It is placed by the deploy, so the same image can run in staging and production with different values.

  • Whole: -rw------- 1 deploy webapp 412 Sep 5 09:41 /srv/app/.env
  • -rw------- — Type and mode: Leading `-` is a regular file (`d` is a directory). Then three triples: owner `rw-`, group `---`, other `---`. Numerically `600`. Nobody but the owner can read this file, which is the whole point of it.
  • deploy — Owner: The user the file belongs to. gunicorn runs as this user, so the owner bits are the ones that apply to it — and to nothing else on the box.
  • webapp — Group: The group bits here are `---`, so group membership grants nothing. Set group read (`640`) only when another service — a log shipper, nginx — genuinely needs it.
  • 412 — Size in bytes: Small, because it holds settings and not data. A `.env` that has grown to kilobytes usually means application data has leaked into configuration.
  • /srv/app/.env — Path: Outside the repository checkout and outside the container image. It is placed by the deploy, so the same image can run in staging and production with different values.

The modes a Django deployment actually uses

The modes a Django deployment actually uses
ModeSymbolicUse it for
`600``-rw-------`the secrets file — owner only, nobody else on the box
`640``-rw-r-----`a log file the shipper's group may read but not write
`644``-rw-r--r--`collected static files served by nginx as another user
`750``drwxr-x---`the application directory — group may enter and read
`755``drwxr-xr-x`a directory nginx must traverse to reach static files
`775` / `777``drwxrwxr-x` / `drwxrwxrwx`**almost never** — `777` on media is how uploads become executable

Together

bash
install -o deploy -g webapp -m 600 /dev/null /srv/app/.env
chown -R deploy:webapp /srv/app && chmod 750 /srv/app

Where configuration can live, and what each choice costs

Where configuration can live, and what each choice costs
HomeVisible toCost
hard-coded in `settings.py`everyone with repository accessa leak is permanent — git history keeps it
a `.env` file, mode `600`the deploy user onlymust be placed by the deploy, never baked into the image
process environmentthat user, via `/proc/<pid>/environ`a restart is needed to change one
a secrets managerwhoever the IAM policy allowsa runtime dependency, and a cache to design

Together

python
import os

SECRET_KEY = os.environ["DJANGO_SECRET_KEY"]   # KeyError at boot beats a default
DEBUG = os.environ.get("DJANGO_DEBUG", "") == "1"

Remember: Permissions are checked owner → group → other, first match wins, and `x` on a directory means "may traverse", which is why `chmod -R 644` breaks a static tree. Run the app as a dedicated `nologin` system user, own the tree as that user at `750`, keep the secrets file at `600`, and never make an upload directory writable and executable. Pass configuration through the environment so one image runs everywhere — but never bake a secret into an `ENV` layer or a committed file, because both are permanent. And test access as the app user with `sudo -u`: as root, every check passes.

See also: systemd the journal and cron · configuration strategy · dependencies secrets and deployment checks · upload validation as untrusted input

Advertisement

Looking at a box

Five diagnostic tools by question, and six text and transfer tools by task.

Looking at a running box: ps, top, htop, lsof, kill

coreintermediate

`ps` prints a **snapshot** of the processes running right now. `top` and `htop` show the same information **continuously**, sorted by whatever is using the most CPU or memory. `lsof` lists open files and sockets, which is how you find out what is listening on a port or which process is holding a deleted log file. `kill` sends a signal to a PID — by default `SIGTERM`, the polite one.

Think of it as

These five tools answer four different questions, and reaching for the wrong one is what makes an incident feel like guesswork. "What is running, and with what arguments?" is `ps` — a snapshot, in the man page's own words, so it is the right tool for structure (which processes exist, who their parent is, what command line they were started with) and the wrong one for watching a number change. "What is consuming the machine right now?" is `top` or `htop`, which re-read the same data every second or two and sort by it; `htop` is `top` with a readable layout, per-core bars, a tree view and the ability to filter and signal from the interface. "Who has this port, this file, this socket?" is `lsof`, and it is the tool people forget they have: `lsof -i :8000` names the process holding the port you cannot bind, and `lsof -p <pid>` shows every file, socket and pipe a worker has open, which is how you catch a leak of file descriptors before it becomes `Too many open files`. "Make it stop" is `kill`, which despite the name simply sends a signal — `kill -TERM` asks, and `kill -KILL` compels. The column that pays for itself is RSS, documented as "the non-swapped physical memory that a task has used": it is real memory for that process, so summing RSS over gunicorn workers is how you learn whether the box is about to start swapping, and comparing RSS across identical workers is how a leak announces itself. VSZ is address space, not memory, and it is routinely large and meaningless; reading VSZ as memory usage is the single most common misdiagnosis on this screen. Two habits make all of this fast. First, filter before you read: `ps -C gunicorn`, `ps --ppid <master>`, `pgrep -f celery` — a full `ps aux` on a busy host is hundreds of lines you will scan wrongly. Second, look at time as well as size: `etime` tells you how long a process has been alive, so a worker with high RSS and three days of uptime is a leak, while the same RSS at four minutes old is just a big request.

bash
ps -o pid,ppid,rss,etime,args -C gunicorn    # snapshot, chosen fields
lsof -i :8000                                # who holds the port

What we're doing: Diagnose "the site is slow and the box is nearly out of memory" without restarting anything first.

triage.sh — read in this orderbash
# Values below are illustrative; the fields are what each tool prints.

# 1. Is the machine itself in trouble? Swap in use means every latency
#    number you are about to read is contaminated by disk I/O.
free -m | awk 'NR==2 {print "used:", $3"MB  free:", $4"MB"}'

# 2. Total resident memory of the web tier, in one number. RSS is KiB
#    and it is the column that means real memory — VSZ is address
#    space and is routinely huge for reasons that do not matter.
ps -o rss= -C gunicorn | awk '{s+=$1} END {print "gunicorn RSS:", s/1024, "MB"}'

# 3. Per worker, with uptime alongside. This is the comparison that
#    identifies a leak: same code, same age, very different RSS.
ps -o pid,rss,etime,stat,args --ppid "$(pgrep -f 'gunicorn: master')"

# 4. Suspect 4814. What is it holding open? A descriptor count that
#    grows with uptime is a socket or file the code never closes.
lsof -p 4814 | wc -l

# 5. Confirm before acting: is the process actually working, or stuck?
#    STAT D is uninterruptible sleep — blocked in the kernel on I/O,
#    which points at the disk or the network, not at Python.
ps -o pid,stat,wchan:20,args -p 4814

# 6. Recycle just that worker's generation, gracefully, via the master.
#    Never kill -9 a worker that is mid-request if TERM would do.
kill -HUP "$(pgrep -f 'gunicorn: master')"
3–5
Start with the machine. If it is swapping, the application looks broken in a dozen ways that are all one symptom, and fixing the application first wastes the outage.
7–10
One number for the whole web tier. This is what you compare against the box's memory to answer "can I add two more workers?" — and the answer is usually about RSS, not about CPU.
12–14
The leak test. Identical workers of identical age with different RSS means a request path is holding memory; the outlier's uptime tells you whether it is a leak or a large request in flight.
16–18
Descriptor count. Every unclosed file, socket and database connection is one; the ceiling is `ulimit -n`, and hitting it produces `OSError: [Errno 24] Too many open files` rather than anything about memory.
20–23
`STAT` distinguishes "busy" from "blocked". `D` is uninterruptible sleep — the process is waiting on the kernel for I/O, so more CPU will not help and `kill -TERM` will not be answered promptly either.
25–27
The action is a graceful worker replacement addressed to the master. It reclaims the leaked memory by exiting the old processes, and no client sees a dropped connection.

Why this works: The diagnosis is ordered from the machine inward, each step narrows the suspect list, and the fix is a graceful replacement rather than a restart that would have destroyed the evidence.

Reading VSZ as memory usage

Wrong

bash
$ ps -o pid,vsz -C gunicorn
  PID    VSZ
 4813 2814920      # "2.8 GB per worker?! we're out of memory"

Better

bash
$ ps -o pid,rss -C gunicorn
  PID    RSS
 4813  189320      # 185 MB actually resident

What you see: A capacity plan built on numbers ten times too large, or an incident review that concludes the app leaks gigabytes when resident memory has been flat all week.

Why: VSZ is "virtual memory size of the process in KiB" — the size of the address space, which includes memory that is mapped but never touched, shared libraries counted in full, and large reservations made by allocators. RSS is "the non-swapped physical memory that a task has used", which is the number that competes for the machine's RAM. They differ by an order of magnitude for a normal Python process, so the two columns support opposite conclusions. Sum RSS to decide how many workers fit; ignore VSZ unless you are debugging address-space exhaustion, which is not what is happening.

Triage order on a box you have just been handed

1 · Is it the app, or the box?

Load average against core count, and the memory line. If the machine is swapping, every other measurement you take will be distorted by it.

2 · Who is consuming it?

Sorted, continuous, per-core. Look for one worker dominating rather than all of them being busy — those are different diagnoses.

3 · Structure, with the fields you care about

Filter to the process family and read RSS against ELAPSED. High RSS plus long uptime is a leak; high RSS plus a young process is one heavy request.

4 · What is it holding?

Open sockets and files for the suspicious PID. A descriptor count that climbs with uptime is the leak that ends in "Too many open files".

5 · Act on the master, not the workers

Graceful first, always. `-9` is for a process that ignored TERM, and it costs you every in-flight request that process was serving.

  1. 1 · Is it the app, or the box? — Load average against core count, and the memory line. If the machine is swapping, every other measurement you take will be distorted by it.
  2. 2 · Who is consuming it? — Sorted, continuous, per-core. Look for one worker dominating rather than all of them being busy — those are different diagnoses.
  3. 3 · Structure, with the fields you care about — Filter to the process family and read RSS against ELAPSED. High RSS plus long uptime is a leak; high RSS plus a young process is one heavy request.
  4. 4 · What is it holding? — Open sockets and files for the suspicious PID. A descriptor count that climbs with uptime is the leak that ends in "Too many open files".
  5. 5 · Act on the master, not the workers — Graceful first, always. `-9` is for a process that ignored TERM, and it costs you every in-flight request that process was serving.

Five tools, four questions

Five tools, four questions
QuestionCommandWhat it gives you
what is running, with what arguments?`ps -o pid,ppid,rss,etime,args -C gunicorn`a snapshot with the fields you chose
what is eating the box right now?`top` / `htop`continuous, sorted by CPU or memory
who holds port 8000?`lsof -i :8000`the process and user behind a bind failure
what has this worker got open?`lsof -p 4814`files, sockets, pipes — the descriptor leak
which processes match this command?`pgrep -a -f "celery worker"`PIDs by command line, not by name
stop it`kill -TERM <pid>`a signal; `-KILL` only when TERM was ignored

Together

bash
ps -o pid,rss,etime,args --ppid "$(pgrep -f 'gunicorn: master')"

The columns, and what they actually mean

The columns, and what they actually mean
ColumnMeaningRead it for
`RSS`resident set size — non-swapped physical memory, in KiBreal memory; sum it, compare it across workers
`VSZ`virtual memory size in KiB — address space**almost nothing**; large values are normal
`%CPU`CPU time over the process's lifetime (in `ps`)nothing live — `top` recomputes it per interval
`STAT`R running, S sleeping, D uninterruptible, Z zombie`D` means blocked on I/O; a wall of `D` is a disk problem
`ELAPSED` / `etime`wall-clock time since the process startedis this RSS a leak, or a four-minute-old request?
`PPID`the parent's PIDreading the tree: which master owns this worker

Together

bash
# RSS is KiB, so /1024 for MB. Total resident memory of the web tier:
ps -o rss= -C gunicorn | awk '{sum += $1} END {print sum/1024 " MB"}'

Remember: `ps` is a snapshot for structure, `top`/`htop` are continuous for consumption, `lsof` answers "who holds this port or file", and `kill` sends a signal rather than killing. Read RSS, never VSZ: RSS is non-swapped physical memory, VSZ is address space and is routinely huge for no reason. Compare RSS across workers of the same age to find a leak, and sum it to decide how many workers fit. Filter before you read, and act on the master with TERM or HUP — `-9` only after TERM was ignored, because it drops everything in flight.

See also: processes pids signals and threads · systemd the journal and cron · dependencies and the tool for each symptom · memory payload and serialization symptoms

grep, awk, sed — and curl, ssh, scp

standardintermediate

`grep` finds the lines that match a pattern. `awk` splits each line into fields and lets you compute with them. `sed` edits a stream of text, usually by substitution. `curl` makes an HTTP request from the command line and can show you the response headers. `ssh` gives you a shell on another machine, and `scp` copies files over that same connection.

Think of it as

These six divide cleanly into "read what happened here" and "reach another machine", and both halves are how you answer questions on a box that has no debugger attached. The text three form a pipeline in increasing power. `grep` selects lines; that is all it does, and it is the right answer for "did this request id appear", "how many 500s in the last rotation", "which settings file mentions ALLOWED_HOSTS". `awk` selects *fields*, so the moment your question involves a column — the fourth field, the status code, the sum of response times — you have moved past `grep`. `sed` transforms, and the honest use for it is substitution in a stream or a file: renaming a setting across a tree, stripping a prefix, redacting a token before you paste a log line into a ticket. There is a house rule that makes all three more useful in a Django context: if your logs are JSON, stop pattern-matching them and parse them. `grep` on a JSON line works until a field value contains the string you are matching, and then it quietly lies; a JSON-aware pass over the same file is both shorter and correct. That is one of the concrete arguments for structured logging. `curl` is the tool that tests what your server actually returns, rather than what your browser renders after ten redirects and a cache. `-i` includes response headers, `-I` sends a HEAD, `-o /dev/null -w` prints timing numbers, and `-H` lets you send the header a proxy would add — which is exactly how you verify that a health check answers without a session, or that `X-Forwarded-Proto` reaches Django. `ssh` is the connection everything else runs over, and the two facts worth internalising are that key authentication should be the only kind enabled, and that `ssh host "command"` runs one command and returns its exit status, which makes it scriptable. `scp` copies over the same transport, so the same host aliases and keys work; for a database dump, streaming it through `ssh` avoids writing the file twice, and for anything repeated `rsync` is the better tool because it resumes.

bash
grep -c '"status": 500' app.log                 # count matching lines
awk -F, '{s += $3} END {print s / NR}' times.csv  # compute over a field
curl -s -o /dev/null -w '%{http_code}\n' URL     # check without the body

What we're doing: Answer "the checkout endpoint got slow at 09:40" from a shell, without deploying anything.

from your laptop, one connectionbash
# 1. Run one command remotely and get its exit status back. This is
#    the shape that works inside a script: no interactive shell, and
#    the "&&" only fires if the service is genuinely active.
ssh web-1 "systemctl is-active gunicorn" && echo "web-1 serving"

# 2. Count the errors in the window. -c counts matching LINES, which
#    is the right unit here because the logger writes one per request.
ssh web-1 "grep -c '\"status\": 500' /var/log/app/app.log"

# 3. Now a field question, so grep is no longer the tool. Pull the
#    duration out of the JSON and average it — jq parses, so a value
#    that happens to contain the word "checkout" cannot fool it.
ssh web-1 "jq -r 'select(.path == \"/checkout/\") | .duration_ms' \
           /var/log/app/app.log | awk '{s+=\$1; n++} END {print s/n, n}'"

# 4. Reproduce it against the app directly, bypassing the proxy, and
#    send the header the proxy would have added. Body discarded; only
#    the status and the timing are printed.
curl -s -o /dev/null -w 'status=%{http_code} total=%{time_total}s\n' \
     -H 'X-Forwarded-Proto: https' https://web-1.internal/checkout/

# 5. Take the slow-query log home. Streaming through ssh avoids
#    writing an intermediate file on a box that may be short of disk.
ssh web-1 "gzip -c /var/log/postgresql/slow.log" > slow.log.gz

# 6. Redact before it goes in a ticket. sed edits the stream, so the
#    original file on the server is untouched.
sed -E 's/(Bearer )[A-Za-z0-9._-]+/\1REDACTED/g' notes.txt > notes.public.txt
1–4
`ssh host "command"` is not an interactive session: it runs one command, prints its output, and exits with its status. That is why it composes with `&&` and can live inside a deploy script.
6–8
`grep -c` counts lines, not matches — the distinction matters when a line can contain the pattern twice. Here one log line is one request, so lines are the unit you want.
10–14
The moment the question involves a field, `grep` stops being correct. `jq` parses the JSON so a match is a match on the *field*, and `awk` does the arithmetic over the values it emits.
16–20
`-o /dev/null -w` measures without printing the body, and `-H` supplies the header the reverse proxy adds — which is how you test the path Django will actually see in production.
22–24
Compressing on the far side and streaming the bytes means nothing is written to the remote disk. `scp` would need the file to exist there first.
26–28
`sed -E` with a capture group replaces the token and keeps the prefix. Redacting before sharing is a habit, not a one-off — a token pasted into a ticket is a token that has been disclosed.

Why this works: Every step is one command from a laptop: service state, an error count, a computed average over a parsed field, a timed reproduction with the right header, a file moved without touching remote disk, and a redaction before sharing.

One question, four tools, in the order they compose

the log file

one line per request, on the box

grep — select lines

only the 500s, only this request id

awk — select fields

the duration column, summed or sorted

sed — transform

redact the token before pasting it anywhere

curl — reproduce it

same path, same headers, from the shell

  1. the log file — one line per request, on the box
  2. grep — select lines — only the 500s, only this request id
  3. awk — select fields — the duration column, summed or sorted
  4. sed — transform — redact the token before pasting it anywhere
  5. curl — reproduce it — same path, same headers, from the shell

The six, with the question each answers on a Django box

The six, with the question each answers on a Django box
ToolQuestion it answersExample
`grep`did this appear, and how often?`grep -c '"status": 500' app.log`
`awk`what do the numbers in this column add up to?`awk '{s+=$NF} END {print s/NR}' times.txt`
`sed`replace this text everywhere`sed -i 's/DEBUG = True/DEBUG = False/' settings.py`
`curl`what does the server really return?`curl -i https://example.com/healthz`
`ssh`run this over there and tell me if it worked`ssh web-1 "systemctl is-active gunicorn"`
`scp`move this file between here and there`scp web-1:/srv/app/dump.sql.gz ./`

Together

bash
# The 20 slowest requests in a plain-text access log, by the last field.
grep " 200 " access.log | awk '{print $NF, $7}' | sort -rn | head -20

curl flags worth knowing by heart

curl flags worth knowing by heart
FlagDoesUse it to
`-i`include response headers in the outputcheck `Set-Cookie`, `Cache-Control`, `Location`
`-I`send a HEAD requesttest a URL without downloading a 40 MB export
`-H`send a headersimulate `X-Forwarded-Proto: https` from the proxy
`-L`follow redirectsconfirm the HTTP → HTTPS chain ends where you think
`-o /dev/null -w`discard the body, print chosen variables`%{http_code}`, `%{time_total}`, `%{size_download}`
`-s`silent — no progress meteranything inside a script or a pipe

Together

bash
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' \
     -H 'X-Forwarded-Proto: https' http://127.0.0.1:8000/readyz

Remember: `grep` selects lines, `awk` selects fields, `sed` transforms — move up only when the question demands it, and on JSON logs parse instead of pattern-matching, because `grep` will match inside a value and quietly mislead you. `curl -i` shows what the server really returned, `-H` lets you send the header the proxy would have added, and `-o /dev/null -w` measures without downloading. `ssh host "cmd"` runs one command and hands back its exit status, which is what makes it scriptable; stream through it rather than writing a file you then `scp`.

See also: looking at a running box · systemd the journal and cron · json logs and context fields · request ids correlation ids and error tracking

Advertisement

What starts things without you

systemd units, the journal, and a cron environment that is emptier than your shell.

systemctl, journalctl and cron

coreintermediate

**systemd** starts your services, restarts them when they die, and starts them again after a reboot. `systemctl` is how you talk to it — start, stop, reload, and ask what state a unit is in. `journalctl` reads the logs systemd collected, filtered by unit and by time. **cron** runs a command on a schedule, and it runs it in a much emptier environment than your shell.

Think of it as

systemd is the answer to "what starts gunicorn, and what starts it again at 03:00 when it dies". A *unit* file describes the service: which command, as which user, with which environment file, and what to do when it exits. Once that exists, the operational vocabulary is small and worth knowing exactly. `start` and `stop` are obvious. `restart` stops and starts, which drops connections. `reload` sends the signal the unit declares for reloading — for gunicorn that is `SIGHUP`, which replaces workers without closing the listening socket, so `reload` is the command you want for a code change and `restart` is the one you want after editing the unit itself. `enable` is the one people forget: it only marks the unit to start at boot, and does nothing right now, which is why "I enabled it but it is not running" is a normal sentence. `status` prints the current state plus the last few log lines, which is usually enough to see why a start failed. The `Restart=` directive is the whole reason to bother — with `Restart=always`, a worker that segfaults comes back without a human, and `RestartSec=` stops a crash loop from becoming a busy loop. Logs then arrive in the journal rather than in a file you have to rotate. `journalctl -u gunicorn -f` is the shape you will type most: `-u` narrows to a unit, `-f` follows new entries as they are appended, and `--since "10 min ago"` bounds a window when you already know when the incident was. Priority filtering (`-p err`) is how you skip the noise. The journal is per-boot and size-capped, so it is for the last hours and days, not for retention — shipping to a central store is a separate concern that the observability section owns. cron is the third piece and the one that fails in a characteristic way. A cron job does not run in your shell: it gets a minimal environment, usually no `PATH` beyond a couple of directories, no virtualenv, no `.bashrc`, and its working directory is the user's home rather than your project. So the command that works when you paste it into a terminal fails at 02:00 with `python: command not found` or `ImproperlyConfigured`. The fix is to make the cron line self-contained: absolute paths to the interpreter and the manage script, the settings module set explicitly, and output redirected somewhere you will actually read. On a systemd box the better answer is often a timer unit instead, because it inherits the same `User=`, `EnvironmentFile=` and journal integration the service already has — one description of the environment rather than two.

bash
systemctl reload gunicorn          # HUP — replace workers, keep the socket
journalctl -u gunicorn -f          # follow that unit's log

What we're doing: Run Django under systemd with automatic restart, and schedule a nightly command that will not fail because of a missing environment.

/etc/systemd/system/gunicorn.service + the crontab linetext
[Unit]
Description=gunicorn for the storefront
After=network.target postgresql.service

[Service]
# Never root. This is the account an application bug inherits.
User=deploy
Group=webapp
WorkingDirectory=/srv/app

# One place the secrets live, readable only by deploy (mode 600).
EnvironmentFile=/srv/app/.env

ExecStart=/srv/app/.venv/bin/gunicorn config.wsgi:application \
          --bind unix:/run/gunicorn.sock --workers 5

# systemctl reload -> SIGHUP -> workers replaced, socket kept open.
ExecReload=/bin/kill -s HUP $MAINPID

# Crash recovery, bounded so a crash loop is not a busy loop.
Restart=always
RestartSec=5

# Give the app longer than its own graceful timeout to finish requests,
# or systemd is the thing sending SIGKILL to a draining worker.
TimeoutStopSec=45

[Install]
WantedBy=multi-user.target


# --- the crontab line (crontab -e as the deploy user) ------------------
# Absolute paths everywhere: cron has no virtualenv and almost no PATH.
0 3 * * * DJANGO_SETTINGS_MODULE=config.settings.production \
  /srv/app/.venv/bin/python /srv/app/manage.py clearsessions \
  >> /var/log/app/cron.log 2>&1
6–8
`User=` is the security boundary. Everything the process can reach — files, sockets, other services — is bounded by this account, so it is the highest-value line in the unit.
11–12
`EnvironmentFile` is how the secret reaches the process without being in the image or the repository. The file is mode `600` and owned by `deploy`, so systemd reads it as root and hands it to a process that runs as `deploy`.
17–18
`ExecReload` is what makes `systemctl reload` mean something. Without it, systemd has no reload action and every deploy becomes a `restart`, which closes the socket and drops connections.
20–22
`Restart=always` covers the crash you have not predicted; `RestartSec=5` is what stops an unbootable configuration from spinning at full CPU.
24–26
This number must exceed gunicorn's own `--graceful-timeout`. If it does not, systemd sends `SIGKILL` while workers are still draining, and the graceful shutdown you configured never completes.
32–36
Every cron failure is one of these three: no interpreter on `PATH`, no settings module, or output going nowhere. Absolute interpreter path, explicit `DJANGO_SETTINGS_MODULE`, and `>>` with `2>&1` remove all three.

Why this works: The service starts at boot as an unprivileged user, reloads without dropping connections, restarts itself after a crash, and the scheduled job carries its own environment instead of borrowing yours.

A cron line that works when you paste it into a shell

Wrong

bash
0 3 * * * cd /srv/app && python manage.py clearsessions
# 02:59 tomorrow: "python: command not found", and nobody is told

Better

bash
0 3 * * * DJANGO_SETTINGS_MODULE=config.settings.production \
  /srv/app/.venv/bin/python /srv/app/manage.py clearsessions \
  >> /var/log/app/cron.log 2>&1

What you see: The job silently never runs. Sessions accumulate for months, the table grows to millions of rows, and the first sign of trouble is a slow query on a model nobody has thought about.

Why: cron does not start a login shell, so it has none of what makes the interactive command work: no activated virtualenv, a `PATH` with two or three entries, no `.bashrc`, and the user's home as the working directory. `python` therefore resolves to the system interpreter or to nothing at all, and Django cannot find its settings. Worse, the failure is silent by default — cron mails the output to a local mailbox nobody reads. Making the line self-contained fixes the first problem and redirecting to a log file fixes the second; a systemd timer fixes both at once by reusing the service unit's `User=` and `EnvironmentFile=`.

A code deploy, through systemd, seen in the journal
you
systemd
gunicorn master
journal
  1. 1. systemctl reload gunicorn
  2. 2. ExecReload → SIGHUPthe listening socket is never closed
  3. 3. Handling signal: hup
  4. 4. Booting worker with pid: 4901new workers start before old ones stop
  5. 5. Worker exiting (pid: 4813)
  6. 6. journalctl -u gunicorn -fyou watch the replacement, live
  7. 7. still running, same main PID
  1. you → systemd: systemctl reload gunicorn
  2. systemd → gunicorn master: ExecReload → SIGHUP (the listening socket is never closed)
  3. gunicorn master → journal: Handling signal: hup
  4. gunicorn master → journal: Booting worker with pid: 4901 (new workers start before old ones stop)
  5. gunicorn master → journal: Worker exiting (pid: 4813)
  6. journal → you: journalctl -u gunicorn -f (you watch the replacement, live)
  7. gunicorn master → systemd: still running, same main PID

The systemctl verbs, and when each is the right one

The systemctl verbs, and when each is the right one
CommandWhat it doesUse it when
`systemctl status gunicorn`state, main PID, and recent log linesfirst command after anything looks wrong
`systemctl reload gunicorn`sends the unit's `ExecReload` signalnew code or config — no dropped connections
`systemctl restart gunicorn`full stop then startyou changed the unit file itself
`systemctl enable --now gunicorn`start now **and** at bootfirst install — `enable` alone starts nothing
`systemctl daemon-reload`re-reads unit files from diskafter editing any `.service` file
`systemctl is-active gunicorn`prints the state, exits non-zero if not activeinside a script or a health check

Together

bash
sudo systemctl daemon-reload        # you edited the unit
sudo systemctl reload gunicorn      # you edited the code

journalctl, the four flags that matter

journalctl, the four flags that matter
FlagMeaningTypical use
`-u <unit>`messages for that systemd unit`journalctl -u celery-worker`
`-f`follow — print new entries as they arrivewatching a deploy land
`-n <N>`the most recent N entries`-n 200` after a failed start
`--since` / `--until`bound the time range`--since "09:35" --until "09:50"`
`-p err`this priority or more importantskipping INFO during an incident

Together

bash
journalctl -u gunicorn -p err --since "30 min ago" -n 100

The five cron fields

The five cron fields
FieldRangeExample value
minute0–59`*/15` — every quarter hour
hour0–23`3` — 03:00
day of month1–31`1` — the first
month1–12`*` — every month
day of week0–7 (0 and 7 are Sunday)`1-5` — weekdays

Together

bash
# min hour dom mon dow   command
    0    3   *   *   *   /srv/app/bin/nightly.sh   # 03:00 daily
  */5    *   *   *   *   /srv/app/bin/poll.sh      # every five minutes

Remember: systemd owns start, restart-on-crash and start-at-boot; `enable` only affects the next boot, so `enable --now` is what you want on install. Use `reload` for code (gunicorn's `ExecReload` sends `SIGHUP` and keeps the socket) and `restart` only after editing the unit. Set `TimeoutStopSec` above your graceful timeout, or systemd is the thing killing draining workers. Read logs with `journalctl -u <unit> -f`, narrowing by `--since` and `-p`; the journal is recent history, not retention. And write cron lines as if nothing is set up — absolute interpreter path, explicit settings module, output redirected — or use a systemd timer and inherit the service's environment.

See also: processes pids signals and threads · users groups permissions and environment · what commands are for · bulk scheduled and external work

Advertisement