Linux for Django Developers cheat sheet · Django

1.Processes, PIDs, signals and threads

A **process** is one running program with its own memory. Its **PID** is the number the kernel gives it, and the number every tool asks you for. A **signal** is a one-byte message you send to a process — `SIGTERM` asks it to stop, `SIGKILL` makes the kernel stop it. A **thread** runs inside a process and shares that process's memory, which is why threads are cheaper and why one thread's crash can take the whole process with it.

Key features

One process = one memory space; workers share nothing but the socket
A PID is kernel-assigned, reused after exit, and different after every restart
SIGTERM (15) asks; SIGKILL (9) "cannot be caught, blocked, or ignored"
A KILLed worker loses its in-flight requests — no cleanup code runs at all
SIGHUP is the conventional "reload config"; gunicorn respawns workers on it
Threads share memory and, under the GIL, buy I/O concurrency — not CPU parallelism

The signals worth knowing, and what gunicorn does with each

The signals worth knowing, and what gunicorn does with each
Signal	No.	Default action	What gunicorn's master does
`SIGTERM`	15	Term	graceful shutdown, up to `graceful_timeout`
`SIGINT` / `SIGQUIT`	2 / 3	Term / Core	quick shutdown
`SIGHUP`	1	Term	reload config, spawn new workers, stop old ones
`SIGTTIN` / `SIGTTOU`	21 / 22	Stop	increase / decrease the worker count by one
`SIGUSR1`	10	Term	reopen log files (what logrotate sends)
`SIGKILL`	9	Term	nothing — it cannot be caught; the kernel stops it

Process or thread — what you actually get

Process or thread — what you actually get
Dimension	Extra process (worker)	Extra thread
memory	a full copy — RSS × N	shared with its process
a crash takes down	that worker only; the master respawns it	usually the whole process
DB connections	one set per process	one per thread, from the same process
CPU-bound work	genuinely parallel	serialised by the GIL
I/O-bound work	parallel	concurrent — this is the case threads help

The process tree a Django deployment actually is

Every worker is a separate process with its own memory and its own database connections. Signals go to the master, which is what makes a graceful restart possible at all.

Command

bash

kill -TERM <pid>     # ask; the process may clean up
kill -KILL <pid>     # the kernel stops it; nothing runs, nothing is saved

A Django deployment is a process tree: systemd → gunicorn master → N workers, each with its own memory and its own database connections — which is why connection count follows process count, not traffic. PIDs are kernel-assigned and reused, so never store one. Signal the master, not the workers: `SIGTERM` drains, `SIGHUP` reloads, and `SIGKILL` (9) "cannot be caught, blocked, or ignored", so it loses every in-flight request. Threads share memory and, under the GIL, buy concurrency while waiting on I/O — not parallel CPU.

2.Users, groups, permissions and the environment

Every file has an **owner**, a **group**, and three sets of permissions — read, write, execute — for the owner, the group, and everyone else. Every process runs *as* a user, and that is what decides which files it may touch. **Environment variables** are name/value pairs the process inherits when it starts; they are where a Django deployment keeps its secrets, because they are not in the repository.

Key features

Mode is checked in order: owner, then group, then other — the first match wins
r=4, w=2, x=1, added per column: `640` = owner rw, group r, other nothing
On a directory, `x` means "may traverse into it" — without it, nothing inside opens
Run the app as a dedicated unprivileged user; root is the blast radius, not a convenience
Secrets file: owned by the deploy user, mode `600`, never in the image or git
Environment variables are inherited at start-up — changing one needs a restart, not a reload

The modes a Django deployment actually uses

The modes a Django deployment actually uses
Mode	Symbolic	Use it for
`600`	`-rw-------`	the secrets file — owner only, nobody else on the box
`640`	`-rw-r-----`	a log file the shipper's group may read but not write
`644`	`-rw-r--r--`	collected static files served by nginx as another user
`750`	`drwxr-x---`	the application directory — group may enter and read
`755`	`drwxr-xr-x`	a directory nginx must traverse to reach static files
`775` / `777`	`drwxrwxr-x` / `drwxrwxrwx`	almost never — `777` on media is how uploads become executable

Where configuration can live, and what each choice costs

Where configuration can live, and what each choice costs
Home	Visible to	Cost
hard-coded in `settings.py`	everyone with repository access	a leak is permanent — git history keeps it
a `.env` file, mode `600`	the deploy user only	must be placed by the deploy, never baked into the image
process environment	that user, via `/proc/<pid>/environ`	a restart is needed to change one
a secrets manager	whoever the IAM policy allows	a runtime dependency, and a cache to design

One line of `ls -l`, field by field

-rw------- 1 deploy webapp 412 Sep 5 09:41 /srv/app/.env

-rw-------

Type and mode — Leading `-` is a regular file (`d` is a directory). Then three triples: owner `rw-`, group `---`, other `---`. Numerically `600`. Nobody but the owner can read this file, which is the whole point of it.

deploy

Owner — The user the file belongs to. gunicorn runs as this user, so the owner bits are the ones that apply to it — and to nothing else on the box.

webapp

Group — The group bits here are `---`, so group membership grants nothing. Set group read (`640`) only when another service — a log shipper, nginx — genuinely needs it.

412

Size in bytes — Small, because it holds settings and not data. A `.env` that has grown to kilobytes usually means application data has leaked into configuration.

/srv/app/.env

Path — Outside the repository checkout and outside the container image. It is placed by the deploy, so the same image can run in staging and production with different values.

Command

bash

chown deploy:webapp /srv/app/.env && chmod 600 /srv/app/.env

Permissions are checked owner → group → other, first match wins, and `x` on a directory means "may traverse", which is why `chmod -R 644` breaks a static tree. Run the app as a dedicated `nologin` system user, own the tree as that user at `750`, keep the secrets file at `600`, and never make an upload directory writable and executable. Pass configuration through the environment so one image runs everywhere — but never bake a secret into an `ENV` layer or a committed file, because both are permanent. And test access as the app user with `sudo -u`: as root, every check passes.

3.Looking at a running box: ps, top, htop, lsof, kill

`ps` prints a **snapshot** of the processes running right now. `top` and `htop` show the same information **continuously**, sorted by whatever is using the most CPU or memory. `lsof` lists open files and sockets, which is how you find out what is listening on a port or which process is holding a deleted log file. `kill` sends a signal to a PID — by default `SIGTERM`, the polite one.

Key features

`ps` is a snapshot; `top`/`htop` are the same data re-read continuously
RSS is real resident memory; VSZ is address space and is not what you want
Sum RSS across workers to predict swapping; compare RSS across workers to spot a leak
`lsof -i :8000` names whoever holds a port; `lsof -p` lists one process's descriptors
`kill` sends a signal, it does not kill — `-9` is the one that cannot be refused
Filter first (`-C`, `--ppid`, `pgrep`); a bare `ps aux` is a list you will misread

Five tools, four questions

Five tools, four questions
Question	Command	What it gives you
what is running, with what arguments?	`ps -o pid,ppid,rss,etime,args -C gunicorn`	a snapshot with the fields you chose
what is eating the box right now?	`top` / `htop`	continuous, sorted by CPU or memory
who holds port 8000?	`lsof -i :8000`	the process and user behind a bind failure
what has this worker got open?	`lsof -p 4814`	files, sockets, pipes — the descriptor leak
which processes match this command?	`pgrep -a -f "celery worker"`	PIDs by command line, not by name
stop it	`kill -TERM <pid>`	a signal; `-KILL` only when TERM was ignored

The columns, and what they actually mean

The columns, and what they actually mean
Column	Meaning	Read it for
`RSS`	resident set size — non-swapped physical memory, in KiB	real memory; sum it, compare it across workers
`VSZ`	virtual memory size in KiB — address space	almost nothing; large values are normal
`%CPU`	CPU time over the process's lifetime (in `ps`)	nothing live — `top` recomputes it per interval
`STAT`	R running, S sleeping, D uninterruptible, Z zombie	`D` means blocked on I/O; a wall of `D` is a disk problem
`ELAPSED` / `etime`	wall-clock time since the process started	is this RSS a leak, or a four-minute-old request?
`PPID`	the parent's PID	reading the tree: which master owns this worker

Triage order on a box you have just been handed

1 · Is it the app, or the box?

Load average against core count, and the memory line. If the machine is swapping, every other measurement you take will be distorted by it.

2 · Who is consuming it?

Sorted, continuous, per-core. Look for one worker dominating rather than all of them being busy — those are different diagnoses.

3 · Structure, with the fields you care about

Filter to the process family and read RSS against ELAPSED. High RSS plus long uptime is a leak; high RSS plus a young process is one heavy request.

4 · What is it holding?

Open sockets and files for the suspicious PID. A descriptor count that climbs with uptime is the leak that ends in "Too many open files".

5 · Act on the master, not the workers

Graceful first, always. `-9` is for a process that ignored TERM, and it costs you every in-flight request that process was serving.

Command

bash

ps -o pid,ppid,rss,etime,args -C gunicorn    # snapshot, chosen fields
lsof -i :8000                                # who holds the port

`ps` is a snapshot for structure, `top`/`htop` are continuous for consumption, `lsof` answers "who holds this port or file", and `kill` sends a signal rather than killing. Read RSS, never VSZ: RSS is non-swapped physical memory, VSZ is address space and is routinely huge for no reason. Compare RSS across workers of the same age to find a leak, and sum it to decide how many workers fit. Filter before you read, and act on the master with TERM or HUP — `-9` only after TERM was ignored, because it drops everything in flight.

4.grep, awk, sed — and curl, ssh, scp

`grep` finds the lines that match a pattern. `awk` splits each line into fields and lets you compute with them. `sed` edits a stream of text, usually by substitution. `curl` makes an HTTP request from the command line and can show you the response headers. `ssh` gives you a shell on another machine, and `scp` copies files over that same connection.

Key features

`grep` selects lines, `awk` selects fields, `sed` transforms text — in that order of power
On JSON logs, parse rather than pattern-match: `grep` matches inside values and lies
`curl -i` shows response headers; `-H` sends the one a proxy would have added
`curl -o /dev/null -w "%{time_total}"` measures without printing the body
`ssh host "cmd"` runs one command and returns its exit status — that is what makes it scriptable
Streaming a dump through `ssh` beats `scp` of a file you had to write to disk first

The six, with the question each answers on a Django box

The six, with the question each answers on a Django box
Tool	Question it answers	Example
`grep`	did this appear, and how often?	`grep -c '"status": 500' app.log`
`awk`	what do the numbers in this column add up to?	`awk '{s+=$NF} END {print s/NR}' times.txt`
`sed`	replace this text everywhere	`sed -i 's/DEBUG = True/DEBUG = False/' settings.py`
`curl`	what does the server really return?	`curl -i https://example.com/healthz`
`ssh`	run this over there and tell me if it worked	`ssh web-1 "systemctl is-active gunicorn"`
`scp`	move this file between here and there	`scp web-1:/srv/app/dump.sql.gz ./`

curl flags worth knowing by heart

curl flags worth knowing by heart
Flag	Does	Use it to
`-i`	include response headers in the output	check `Set-Cookie`, `Cache-Control`, `Location`
`-I`	send a HEAD request	test a URL without downloading a 40 MB export
`-H`	send a header	simulate `X-Forwarded-Proto: https` from the proxy
`-L`	follow redirects	confirm the HTTP → HTTPS chain ends where you think
`-o /dev/null -w`	discard the body, print chosen variables	`%{http_code}`, `%{time_total}`, `%{size_download}`
`-s`	silent — no progress meter	anything inside a script or a pipe

One question, four tools, in the order they compose

the log file

one line per request, on the box

grep — select lines

only the 500s, only this request id

awk — select fields

the duration column, summed or sorted

sed — transform

redact the token before pasting it anywhere

curl — reproduce it

same path, same headers, from the shell

Command

bash

grep -c '"status": 500' app.log                 # count matching lines
awk -F, '{s += $3} END {print s / NR}' times.csv  # compute over a field
curl -s -o /dev/null -w '%{http_code}\n' URL     # check without the body

`grep` selects lines, `awk` selects fields, `sed` transforms — move up only when the question demands it, and on JSON logs parse instead of pattern-matching, because `grep` will match inside a value and quietly mislead you. `curl -i` shows what the server really returned, `-H` lets you send the header the proxy would have added, and `-o /dev/null -w` measures without downloading. `ssh host "cmd"` runs one command and hands back its exit status, which is what makes it scriptable; stream through it rather than writing a file you then `scp`.

5.systemctl, journalctl and cron

**systemd** starts your services, restarts them when they die, and starts them again after a reboot. `systemctl` is how you talk to it — start, stop, reload, and ask what state a unit is in. `journalctl` reads the logs systemd collected, filtered by unit and by time. **cron** runs a command on a schedule, and it runs it in a much emptier environment than your shell.

Key features

`enable` only affects the next boot; `start` is what runs it now
`reload` sends the unit's reload signal (HUP for gunicorn); `restart` drops connections
`Restart=always` plus `RestartSec=` is what makes a crash self-healing, not a loop
`journalctl -u <unit> -f` follows a single unit; `--since`/`-p` narrow it further
The journal is per-boot and size-capped — it is recent history, not retention
cron runs with a minimal environment: absolute paths, explicit settings, redirected output

The systemctl verbs, and when each is the right one

The systemctl verbs, and when each is the right one
Command	What it does	Use it when
`systemctl status gunicorn`	state, main PID, and recent log lines	first command after anything looks wrong
`systemctl reload gunicorn`	sends the unit's `ExecReload` signal	new code or config — no dropped connections
`systemctl restart gunicorn`	full stop then start	you changed the unit file itself
`systemctl enable --now gunicorn`	start now and at boot	first install — `enable` alone starts nothing
`systemctl daemon-reload`	re-reads unit files from disk	after editing any `.service` file
`systemctl is-active gunicorn`	prints the state, exits non-zero if not active	inside a script or a health check

journalctl, the four flags that matter

journalctl, the four flags that matter
Flag	Meaning	Typical use
`-u <unit>`	messages for that systemd unit	`journalctl -u celery-worker`
`-f`	follow — print new entries as they arrive	watching a deploy land
`-n <N>`	the most recent N entries	`-n 200` after a failed start
`--since` / `--until`	bound the time range	`--since "09:35" --until "09:50"`
`-p err`	this priority or more important	skipping INFO during an incident

The five cron fields

The five cron fields
Field	Range	Example value
minute	0–59	`*/15` — every quarter hour
hour	0–23	`3` — 03:00
day of month	1–31	`1` — the first
month	1–12	`*` — every month
day of week	0–7 (0 and 7 are Sunday)	`1-5` — weekdays

A code deploy, through systemd, seen in the journal

you

systemd

gunicorn master

journal

1systemctl reload gunicorn
2ExecReload → SIGHUPthe listening socket is never closed
3Handling signal: hup
4Booting worker with pid: 4901new workers start before old ones stop
5Worker exiting (pid: 4813)
6journalctl -u gunicorn -fyou watch the replacement, live
7still running, same main PID

Command

bash

systemctl reload gunicorn          # HUP — replace workers, keep the socket
journalctl -u gunicorn -f          # follow that unit's log

systemd owns start, restart-on-crash and start-at-boot; `enable` only affects the next boot, so `enable --now` is what you want on install. Use `reload` for code (gunicorn's `ExecReload` sends `SIGHUP` and keeps the socket) and `restart` only after editing the unit. Set `TimeoutStopSec` above your graceful timeout, or systemd is the thing killing draining workers. Read logs with `journalctl -u <unit> -f`, narrowing by `--since` and `-p`; the journal is recent history, not retention. And write cron lines as if nothing is set up — absolute interpreter path, explicit settings module, output redirected — or use a systemd timer and inherit the service's environment.