Filter concepts by levelShowing all levels.

Python · Section 34

Linux

Level
intermediate
Read
190 min
Concepts
8

The Linux process model, signals, file permissions and users, environment variables, and the command-line toolset (ps/top/htop/lsof/kill, grep/awk/sed, curl/wget, ssh/scp, systemd, cron) a backend Python engineer uses to run, debug, and operate code on a real server.

What is true here

  1. subprocess.run([...]) starts a genuinely new process with its own PID — not a thread sharing this one's memory.
  2. kill PID sends SIGTERM (catchable, the polite default); kill -9 PID sends SIGKILL (cannot be caught or ignored).
  3. Every file has an owning user, an owning group, and three read/write/execute triads — owner, group, other.
  4. ps/top/htop find what is running; lsof shows what it has open; kill signals it — one loop for debugging a server.
  5. systemd keeps a service running and restarts it on crash (Restart=on-failure); cron runs a command on a fixed schedule and does not wait for the last run to finish.

What you will be able to do

  • Explain the difference between a process and a thread, and start a real subprocess from Python
  • Send and handle a signal (SIGTERM, SIGINT) correctly for graceful shutdown
  • Read and change file permissions and ownership with chmod/os.chmod and understand the owner/group/other model
  • Use ps, top/htop, lsof, and kill together to find, inspect, and stop a misbehaving process
  • Chain grep, awk, and sed to search, extract, and rewrite text from the command line
  • Use curl to test an endpoint and capture a status code from a script, and know when wget fits better
  • Read a systemd unit file and explain what makes a crashed service restart automatically
  • Write a cron job that will not silently fail from a minimal PATH or swallowed output

The process model, signals, and permissions

What a process is, how a signal reaches one, and the ownership/permission model every file on Linux carries.

Processes, PIDs, and threads

corebeginner

A process is a running program with its own memory space and a unique process ID (PID) the kernel assigns it. A thread runs inside a process and shares that process's memory with any other threads in it.

Think of it as

A process is a separate apartment — its own walls, its own furniture, nothing leaks into the apartment next door unless you deliberately open a door (a pipe, a socket, shared memory). A thread is a roommate inside that same apartment — free movement through every room, but also free to knock over the same furniture another roommate is using at the same moment.

python
import os

os.getpid()    # this process's own PID
os.getppid()   # the PID of the process that started it (its parent)

What we're doing: Read the current process's PID and parent PID, then start a genuinely separate child process with subprocess and confirm it has a different PID.

process_identity.pypython
import os
import subprocess

pid = os.getpid()
ppid = os.getppid()
print(f"this process: PID {pid}, parent PID {ppid}")

result = subprocess.run(
    ["python3", "-c", "import os; print(os.getpid())"],
    capture_output=True, text=True,
)
child_pid = int(result.stdout.strip())
print(f"child process PID: {child_pid}")
print("different PID:", child_pid != pid)
4
os.getpid() returns the PID the kernel assigned to THIS running Python process.
5
os.getppid() returns the PID of whatever process started this one — a shell, or another Python process.
8
subprocess.run([...]) starts a genuinely new process with its own PID and its own memory — not a thread.
Output
this process: PID 8676, parent PID 10520
child process PID: 22040
different PID: True

Why this works: The kernel hands out a fresh, unique PID every time a new process starts — subprocess.run() asks the kernel to create one, so the child's PID is guaranteed different from the parent's. Contrast this with threading.Thread(...): a thread started that way shares the SAME PID as the process that created it, because it is not a separate process at all.

Assuming a thread has its own PID like a process does

Wrong

python
import os, threading

def worker():
    print("worker PID:", os.getpid())   # same PID as the main thread!

t = threading.Thread(target=worker)
t.start()
t.join()
print("main PID:", os.getpid())

Better

python
import os, threading

def worker():
    # os.getpid() is the same everywhere in this process -- use
    # threading.get_ident() to tell threads apart, not PID.
    print("worker thread id:", threading.get_ident(), "in PID:", os.getpid())

t = threading.Thread(target=worker)
t.start()
t.join()
print("main thread id:", threading.get_ident())

What you see: os.getpid() prints the identical number in every thread of the same process — logs that key on PID alone cannot tell which thread produced a given line.

Why: A PID identifies a process, not a thread — every thread inside one process shares that process's single PID because they share its whole memory space, PID included. To distinguish threads in logs or debugging, use threading.get_ident() or threading.current_thread().name instead of os.getpid().

One process, two threads sharing its memory

Process (PID 4821)

its own memory space

Thread 1

reads/writes the process's memory

Thread 2

reads/writes the SAME memory

  1. Process (PID 4821) — its own memory space
  2. Thread 1 — reads/writes the process's memory
  3. Thread 2 — reads/writes the SAME memory

Process vs. thread, and how Python starts each

Process vs. thread, and how Python starts each
AspectProcessThread
Memoryown address space, isolatedshares its process's memory
IdentifierPID, unique system-wideno PID of its own — lives inside one
Crash isolationone process crashing does not take down anotheran unhandled exception in one thread can affect the process
Started from Pythonsubprocess.run(...), multiprocessing.Process(...)threading.Thread(...)
Cost to startheavier — new memory space, new PIDlighter — reuses the parent process's memory

Together

python
import os, subprocess

print("this process PID:", os.getpid())
result = subprocess.run(["python3", "-c", "import os; print(os.getpid())"],
                         capture_output=True, text=True)
print("child process PID:", result.stdout.strip())

Remember: A process has its own memory and a unique PID; a thread runs inside a process and shares that process's memory — subprocess.run() gets you the former, threading.Thread() the latter.

See also: threading and thread · multiprocessing basics · what the gil is · os sys subprocess · signals · process inspection toolset

Signals

coreintermediate

A signal is a short notification the kernel or another process sends to a running process — SIGTERM asks it to stop, SIGKILL forces it to stop. Python's signal module lets a process run its own code when one arrives instead of just dying.

Think of it as

A signal is a knock on the door, not a conversation — it carries no data, only which knock it was (SIGTERM, SIGINT, SIGHUP...). The process can register a handler to answer a specific knock its own way (save state, close files, then exit) — except SIGKILL, which is the landlord using a master key: no answering, no cleanup, the process is gone the instant it arrives.

python
import signal

def handler(signum, frame):
    ...  # signum is the signal number that arrived; frame is the interrupted stack frame

signal.signal(signal.SIGTERM, handler)   # register
signal.signal(signal.SIGTERM, signal.SIG_DFL)   # restore the default action

What we're doing: Register a SIGTERM handler that sets a shutdown flag instead of dying immediately, so a running loop can finish its current unit of work first.

graceful_shutdown.pypython
import signal

shutdown_requested = False

def handle_sigterm(signum, frame):
    global shutdown_requested
    shutdown_requested = True
    print(f"received signal {signum}, finishing current task then stopping")

signal.signal(signal.SIGTERM, handle_sigterm)

# --- simulating delivery for this example (see mistake box below) ---
handle_sigterm(signal.SIGTERM, None)
print("shutdown_requested is now:", shutdown_requested)
9
signal.signal(signal.SIGTERM, handle_sigterm) tells Python to run handle_sigterm whenever this process receives SIGTERM, instead of dying immediately.
5
global shutdown_requested lets the handler mutate a flag the main loop checks — the handler itself should stay short and never do the real cleanup work inline.
12
handle_sigterm is called directly here to show what runs when the signal arrives — a real deployment sends the signal externally (systemctl stop, docker stop, kill PID).
Output
received signal 15, finishing current task then stopping
shutdown_requested is now: True

Why this works: signal.signal(SIGTERM, handler) replaces the default action (immediate termination) with a Python function the interpreter runs when SIGTERM is delivered. Setting a flag instead of exiting inside the handler lets a main loop notice shutdown_requested on its own schedule and finish whatever unit of work is in progress — this is exactly how systemctl stop and docker stop are meant to be handled: they send SIGTERM and wait, then send SIGKILL only if the process has not exited after a timeout.

Doing slow work inside a signal handler

Wrong

python
import signal, time

def handle_sigterm(signum, frame):
    print("shutting down")
    time.sleep(5)          # blocks signal delivery -- risky
    flush_buffers_to_disk()  # slow I/O inside the handler itself

signal.signal(signal.SIGTERM, handle_sigterm)

Better

python
import signal

shutdown_requested = False

def handle_sigterm(signum, frame):
    global shutdown_requested
    shutdown_requested = True   # just set a flag -- fast, safe

signal.signal(signal.SIGTERM, handle_sigterm)

# main loop checks the flag and does the real cleanup on its own time
while not shutdown_requested:
    do_one_unit_of_work()
flush_buffers_to_disk()

What you see: The process hangs during shutdown, or gets SIGKILLed by the orchestrator (Docker, Kubernetes, systemd all send SIGKILL after a grace period, commonly 10-30 seconds) before cleanup finishes.

Why: A signal handler runs at an arbitrary point in the program, and only a small set of operations are guaranteed safe to call from inside one; long-running or blocking work risks reentrancy bugs and eats into the grace period a supervisor gives before escalating to SIGKILL. Setting a flag and letting the main loop react to it keeps the handler itself instant.

A signal interrupts a running process

kill -TERM 4821

another process sends SIGTERM

kernel delivers it

process 4821 is interrupted

handler runs

save state, close files, exit(0)

  1. kill -TERM 4821 — another process sends SIGTERM
  2. kernel delivers it — process 4821 is interrupted
  3. handler runs — save state, close files, exit(0)

Signals a backend engineer actually runs into

Signals a backend engineer actually runs into
SignalNumberDefault actionTypical source
SIGTERM15terminate (catchable)kill PID, systemctl stop, container shutdown
SIGKILL9terminate (NOT catchable)kill -9 PID, OOM killer
SIGINT2terminate (catchable)Ctrl+C in a terminal
SIGHUP1terminate (catchable)terminal closed; daemons repurpose it as "reload config"
SIGCHLD17ignored by defaulta child process exited — the parent is notified

Together

python
import signal

def on_term(signum, frame):
    print("SIGTERM received, shutting down cleanly")
    raise SystemExit(0)

def on_hup(signum, frame):
    print("SIGHUP received, reloading config")

signal.signal(signal.SIGTERM, on_term)
signal.signal(signal.SIGHUP, on_hup)
# SIGKILL has no signal.SIGKILL handler registration that works --
# it cannot be caught. There is nothing to install it against.

Remember: SIGTERM is catchable and asks nicely — use it for cleanup logic; SIGKILL cannot be caught and ends the process immediately, with zero cleanup.

See also: processes and pids · process inspection toolset · finally clause · os sys subprocess

File permissions, users, and groups

standardbeginner

Every Linux file has an owning user, an owning group, and permission bits for each of owner/group/everyone-else — read, write, and execute. chmod 600 secrets.env means only the owner can read or write it, and no one can execute it.

Think of it as

Think of a file's permissions as three separate locks on the same door — one for you (the owner), one for your group, one for everybody else — and each lock has three settings: can open it (read), can rearrange the furniture (write), can walk through it and run what's inside (execute). rwxr-xr-- reads as three of those triads in a row: owner, then group, then other.

python
import os
from pathlib import Path

os.chmod("secrets.env", 0o600)      # owner rw-, nobody else anything
Path("deploy.sh").chmod(0o755)      # owner rwx, group/other rx

What we're doing: Decode common octal permission modes into the rwx string a real ls -l or chmod would show, and confirm the round-trip.

decode_permissions.pypython
import stat

for octal in (0o600, 0o644, 0o755, 0o777):
    mode = stat.S_IFREG | octal
    print(oct(octal), "->", stat.filemode(mode))

# stat.S_IMODE strips the file-type bits back off, recovering just the octal
print(oct(stat.S_IMODE(stat.S_IFREG | 0o640)))
3
stat.S_IFREG | octal builds a full mode value the way the kernel stores it — file type bits plus permission bits.
4
stat.filemode() turns that mode into the same rwx string ls -l or chmod would display.
7
stat.S_IMODE() does the reverse — given a full mode, it strips the file-type bits and returns just the permission bits.
Output
0o600 -> -rw-------
0o644 -> -rw-r--r--
0o755 -> -rwxr-xr-x
0o777 -> -rwxrwxrwx
0o640

Why this works: A file mode is one integer that packs a file-type field and three permission triads together — stat.S_IFREG marks it as a regular file, and the low bits carry owner/group/other permissions exactly the way chmod's octal argument does. stat.filemode() is what turns that packed integer into the human-readable string, and it is the same decoding chmod 600 file and Path(file).chmod(0o600) act on when applied on a real POSIX filesystem.

Permission bits, octal digits, and what they mean

Permission bits, octal digits, and what they mean
SymbolOctalMeaning
r--4read only
-w-2write only
--x1execute (or "enter", for a directory)
rw-6read + write
r-x5read + execute — typical for a directory or a script others may run
rwx7read + write + execute — full access for that triad

Together

python
import stat

# Decode an octal mode into the rwx string chmod/ls -l would show.
for octal in (0o600, 0o644, 0o755):
    mode = stat.S_IFREG | octal
    print(oct(octal), "->", stat.filemode(mode))

Remember: Octal permission = owner digit, group digit, other digit — each digit is read(4)+write(2)+execute(1) added together; chmod 600 means owner-only read/write.

See also: environment variables and filesystem · processes and pids · pathlib module

Environment variables and the filesystem

standardbeginner

An environment variable is a named value passed into a process from its shell or supervisor — os.environ["DB_HOST"] reads one. The filesystem is a single tree rooted at /, with conventional directories like /etc for config and /var/log for logs.

Think of it as

Environment variables are sticky notes handed to a process the moment it starts — DB_HOST=prod-db, LOG_LEVEL=info — readable for its whole life but not something one process can hand to another already running. The filesystem is the one shared map every process reads that layout from — /etc for settings, /var for data that changes, /tmp for anything disposable, /home for a user's own files.

python
import os

os.environ.get("DATABASE_URL", "postgresql://localhost/dev")  # safe, has a default
os.environ["DATABASE_URL"]                                    # raises KeyError if unset

What we're doing: Read a required and an optional environment variable with different failure behavior, and build a config path under the conventional /etc layout with pathlib.

read_config.pypython
import os
from pathlib import PurePosixPath

log_level = os.environ.get("LOG_LEVEL", "INFO")
print("log level:", log_level)

try:
    api_key = os.environ["API_KEY"]
except KeyError:
    raise SystemExit("API_KEY is required but not set in the environment")

config_path = PurePosixPath("/etc") / "myapp" / "config.yaml"
print("config path:", config_path)
4
os.environ.get(..., "INFO") returns "INFO" when LOG_LEVEL is unset instead of raising — appropriate for an optional setting.
8
os.environ["API_KEY"] raises KeyError immediately when the variable is missing — appropriate for a setting the process cannot safely run without.
12
PurePosixPath("/etc") / "myapp" / "config.yaml" builds a real Linux path with / joins, regardless of what OS this code happens to run on.
Output
log level: INFO
Traceback (most recent call last):
  ...
SystemExit: API_KEY is required but not set in the environment

Why this works: os.environ behaves like a dict, so .get() with a default is the right choice for anything that has a sane fallback, while plain indexing is the right choice for anything the process must refuse to start without — failing loudly and immediately beats silently running with a missing API key. PurePosixPath (rather than Path) is used here specifically because it always uses / and Linux path rules no matter what OS runs the check — a plain Path would use backslashes on Windows.

Standard Linux directories a backend engineer references most

Standard Linux directories a backend engineer references most
PathWhat lives there
/etcsystem-wide configuration files (nginx.conf, crontab, ssh config)
/var/loglog files most services write to by convention
/var/libpersistent application state (databases, package manager data)
/tmpscratch space, cleared on reboot — never store anything that must survive
/home/<user>a user's own files and dotfiles (~/.bashrc, ~/.ssh)
/proca virtual filesystem exposing live kernel/process info — not real files on disk

Together

python
from pathlib import PurePosixPath

log_dir = PurePosixPath("/var/log/myapp")
config_file = PurePosixPath("/etc/myapp/config.yaml")
print(log_dir / "access.log")
print(config_file.parent, config_file.name)

Remember: os.environ.get(key, default) for optional config, os.environ[key] for required config that should fail fast — and PurePosixPath when a path must follow Linux rules regardless of host OS.

See also: permissions users and groups · pathlib module · os sys subprocess · environment variables and env files

Advertisement

Inspecting processes and searching text

The five-command loop for debugging a running server, and the three-command toolset for searching and rewriting text on it.

Process inspection: ps, top, htop, lsof, kill

standardbeginner

ps lists processes, top/htop show them live with resource usage, lsof lists the files and network connections a process has open, and kill sends a process a signal — most often to stop it.

Think of it as

These five commands are one loop for debugging a running server: ps or top to find the misbehaving process, lsof to see what it currently has open (files, ports, sockets), and kill to signal it once you know what to do — restart, reload, or stop. Reach for a specific one by the question you are actually asking, not by habit.

python
import subprocess

subprocess.run(["ps", "aux"])                 # BSD-style: no leading dash
subprocess.run(["lsof", "-i", ":8000"])        # what's listening on port 8000
subprocess.run(["kill", "-15", str(pid)])      # SIGTERM, the polite default
subprocess.run(["kill", "-9", str(pid)])       # SIGKILL, cannot be ignored

What we're doing: Run ps -ef from Python with subprocess, and parse its header row and process count from the real captured output.

list_processes.pypython
import subprocess

result = subprocess.run(["ps", "-ef"], capture_output=True, text=True)
header, *rows = result.stdout.splitlines()
print(header)
print(f"{len(rows)} processes listed")
3
capture_output=True, text=True captures stdout as a decoded string instead of letting it print directly to the terminal.
4
result.stdout.splitlines() breaks the output into lines — the first is the column header (ps -ef always prints one).
Output
UID          PID    PPID  C STIME TTY          TIME CMD
15 processes listed

Why this works: ps -ef always writes a fixed header line first (UID PID PPID C STIME TTY TIME CMD on Linux), so splitting the captured stdout on newlines and taking the first line reliably isolates it from the process rows that follow — the same technique subprocess-driven monitoring scripts use to turn ps output into structured data without a dedicated library. (Header shown is real Linux ps -ef per man7.org ps(1); the subprocess/splitlines mechanics were executed against this environment's ps, which returns the same header shape with one fewer column.)

Parsing ps output by splitting on whitespace without limiting the CMD column

Wrong

python
import subprocess

result = subprocess.run(["ps", "-ef"], capture_output=True, text=True)
for line in result.stdout.splitlines()[1:]:
    uid, pid, ppid, c, stime, tty, time, cmd = line.split()
    # ValueError: too many values to unpack -- CMD often contains spaces
    # ("python3 server.py --port 8000" is one field, not four)

Better

python
import subprocess

result = subprocess.run(["ps", "-ef"], capture_output=True, text=True)
for line in result.stdout.splitlines()[1:]:
    # split with maxsplit=7 keeps everything after the 7th gap as one field
    fields = line.split(maxsplit=7)
    pid, cmd = fields[1], fields[7]
    print(pid, cmd)

What you see: ValueError: too many values to unpack — a plain .split() breaks CMD into extra pieces whenever the command itself contains spaces or arguments, which is nearly always.

Why: ps -ef's CMD column is the full command line, including arguments, and is not a single whitespace-free token — an unbounded .split() cannot tell where CMD starts. Passing maxsplit to str.split() caps how many splits happen, leaving the remainder — the whole command line — as one final field.

ps, top, htop, lsof, kill — one row each

ps, top, htop, lsof, kill — one row each
CommandAnswersCommon invocation
ps"what processes exist right now"ps aux or ps -ef
top"what is using CPU/memory right now, live"top (press q to quit)
htop"same as top, more readable"htop (not always preinstalled)
lsof"what files/ports does this process have open"lsof -i :8000 · lsof -p PID
kill"stop (or signal) this process"kill PID · kill -9 PID

Together

python
import subprocess

# Find every process, then filter for one this Python script itself started.
result = subprocess.run(["ps", "-ef"], capture_output=True, text=True)
header, *rows = result.stdout.splitlines()
print(header)
print(f"{len(rows)} processes currently listed")

Remember: ps/top/htop answer "what is running"; lsof answers "what does it have open"; kill sends a signal — SIGTERM (default, catchable) before SIGKILL (-9, not catchable).

See also: processes and pids · signals · os sys subprocess

Text processing: grep, awk, sed

standardbeginner

grep searches text for lines matching a pattern, awk extracts and processes fields from structured lines, and sed rewrites text by pattern — three small tools that read a log file faster than opening it in Python for a one-off check.

Think of it as

These three do one job each, and chain together with a pipe: grep finds the lines you care about, awk pulls specific fields out of them, sed rewrites text in place. Reaching for a five-line Python script to filter a log is usually slower to write than one grep | awk pipeline — save Python for logic these three cannot express.

bash
grep -E 'ERROR|WARN' app.log        # extended regex, either pattern
awk -F',' '{print $2}' data.csv     # -F sets the field separator
sed -i 's/DEBUG/INFO/g' app.log     # -i edits the file in place, g = every match

What we're doing: Filter a log for ERROR lines, extract the user field with awk, and swap a word with sed — chaining all three the way a server debugging session actually does.

debug_log.shbash
# app.log:
# INFO user=alice action=login
# ERROR user=bob action=login msg=timeout
# INFO user=carol action=logout
# ERROR user=alice action=purchase msg=declined

grep ERROR app.log
grep -c ERROR app.log
awk -F'user=' '/ERROR/{split($2,a," "); print a[1]}' app.log
sed 's/ERROR/FAIL/' app.log | sed -n '2p'
7
grep ERROR app.log prints only the two lines containing ERROR, unchanged.
8
grep -c counts matching lines instead of printing them — 2, not the lines themselves.
9
awk -F'user=' splits each line on "user="; on ERROR lines, splitting the second piece on space isolates just the username.
10
sed 's/ERROR/FAIL/' rewrites the word on every line as it streams by; piping into sed -n '2p' then keeps only the (now-rewritten) second line.
Output
ERROR user=bob action=login msg=timeout
ERROR user=alice action=purchase msg=declined
2
bob
alice
FAIL user=bob action=login msg=timeout

Why this works: Each tool narrows the data further than the last: grep's job is entirely to select lines, awk's -F flag changes what it treats as a field separator so unstructured key=value logs still split cleanly, and sed operates as a stream editor — its substitutions apply to every line as it passes through, independent of any earlier grep or awk in the same pipeline. Real GNU grep/awk/sed on this line-per-entry log confirm the exact output above.

Forgetting sed's substitution only replaces the first match per line without the g flag

Wrong

bash
# line: ERROR ERROR ERROR count=3
sed 's/ERROR/FAIL/' app.log
# FAIL ERROR ERROR count=3  -- only the first ERROR was replaced

Better

bash
# line: ERROR ERROR ERROR count=3
sed 's/ERROR/FAIL/g' app.log
# FAIL FAIL FAIL count=3  -- g makes it replace every match on the line

What you see: Only the first occurrence per line changes; every occurrence after it on the same line is left untouched, with no warning.

Why: sed's s/OLD/NEW/ command, without a trailing flag, replaces only the first match it finds on each line — this mirrors str.replace(old, new, 1) in Python, not the no-count default. The g (global) flag is what extends the substitution to every match on that line.

grep, awk, sed — one row each

grep, awk, sed — one row each
CommandJobExample
grepfind lines matching a patterngrep ERROR app.log
grep -ccount matching linesgrep -c ERROR app.log
awkextract/process fields from each lineawk '{print $1, $2}' app.log
sed s/../../substitute textsed 's/ERROR/FAIL/' app.log
sed -n Npprint only line Nsed -n '2p' app.log

Together

bash
grep ERROR app.log | awk '{print $1}' | sort | uniq -c

Remember: grep finds lines, awk extracts fields from a line, sed rewrites text — sed's s/// only replaces the first match per line unless you add g.

See also: processes and pids · process inspection toolset · re module

Advertisement

Transferring data, connecting, and scheduling

Fetching or downloading over HTTP from the shell, reaching a remote machine, and keeping a service running or scheduled.

curl and wget

standardbeginner

curl sends an HTTP request from the shell and prints or captures the response — useful for testing an API or scripting a call. wget is built specifically for downloading a file to disk, with better default retry/resume behavior.

Think of it as

curl is a general-purpose HTTP client you script around — it prints the response to stdout by default so you can pipe it, filter it, or capture it into a variable. wget is a downloader first: point it at a URL and it saves the file, resuming automatically if the connection drops. Reach for curl when you need to inspect or process a response; reach for wget for "get this file onto disk and don't make me babysit it."

bash
curl -s https://httpbin.org/get                 # print the body
curl -sI https://httpbin.org/get                # headers only, no body
curl -s -o out.json https://httpbin.org/get     # save body to out.json
curl -s -w "%{http_code}" -o /dev/null URL      # just the status code

What we're doing: Run curl from Python with subprocess to fetch a real response, then extract just its HTTP status code with -w.

check_status.pypython
import subprocess

result = subprocess.run(
    ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "https://httpbin.org/status/404"],
    capture_output=True, text=True,
)
print("status:", result.stdout)
4
-o /dev/null discards the response body — this call only cares about the status code, not the content.
4
-w "%{http_code}" tells curl to print the numeric status code after the transfer, instead of any body.
Output
status: 404

Why this works: httpbin.org/status/404 always returns a 404 response with an empty body, by design — a fixed endpoint for testing status-code handling. -o /dev/null throws away that empty body, and -w "%{http_code}" substitutes the real status curl received, so the script prints just the number a caller actually needs to branch on.

Checking curl success by whether it printed anything, not its actual exit code or status

Wrong

python
import subprocess

result = subprocess.run(["curl", "-s", "https://httpbin.org/status/500"], capture_output=True, text=True)
if result.stdout:
    print("looked fine")   # wrong: a 500 response can still have a non-empty body

Better

python
import subprocess

result = subprocess.run(
    ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "https://httpbin.org/status/500"],
    capture_output=True, text=True,
)
if result.stdout.startswith("2"):
    print("actually succeeded")
else:
    print("failed with status", result.stdout)

What you see: A script reports success on a 4xx/5xx response because curl still exits 0 and prints a body by default — curl does not fail its own exit code just because the SERVER returned an error status.

Why: By default curl only fails (nonzero exit code) on a connection-level problem — DNS failure, refused connection, timeout — not on an HTTP error status. A 404 or 500 is still a "successful" HTTP exchange from curl's point of view. Reading the real status via -w (or adding --fail, which makes curl itself exit nonzero on 4xx/5xx) is the only reliable check.

curl vs. wget for the same task

curl vs. wget for the same task
Taskcurlwget
Print a response bodycurl -s URLwget -qO- URL
Download to a filecurl -O URLwget URL
Headers onlycurl -I URLwget --spider -S URL
Resume a partial downloadcurl -C - -O URLwget -c URL
Send JSON with a methodcurl -X POST -d '{"a":1}' URLwget --method=POST --body-data='{"a":1}' URL

Together

python
import subprocess

result = subprocess.run(
    ["curl", "-s", "-w", "%{http_code}", "-o", "/dev/null", "https://httpbin.org/status/404"],
    capture_output=True, text=True,
)
print("status:", result.stdout)

Remember: curl -s prints the response and is scriptable; curl -w "%{http_code}" -o /dev/null gets just the status. wget defaults to resumable file downloads; curl needs -C - to do the same.

See also: text processing toolset · client library landscape

Remote access and scheduling: ssh, scp, systemd, cron

standardbeginner

ssh opens a remote shell on another machine over an encrypted connection; scp copies files to or from that machine using the same connection. systemd keeps a service process running and restarts it on crash or reboot; cron runs a command on a fixed schedule.

Think of it as

ssh and scp are the same underlying encrypted connection used two ways — one for an interactive shell, one for moving files. systemd and cron solve a different pair of problems: systemd keeps something ALWAYS running (a web server, a worker) and restarts it if it dies; cron runs something PERIODICALLY (a nightly backup, an hourly report) and does not care if the last run is still finishing.

bash
ssh user@host                              # interactive remote shell
ssh user@host "ls -la /srv/app"            # run one command, then exit
scp local.txt user@host:/remote/path/      # copy a file up
scp user@host:/remote/file.txt .           # copy a file down

systemctl status myapp.service             # is it running, and why not
systemctl restart myapp.service            # restart it now

crontab -l                                 # list the current user's cron jobs
crontab -e                                 # edit them

What we're doing: Read a systemd unit file that keeps a Python service running and restarts it automatically on crash.

myapp.servicetoml
[Unit]
Description=My backend service
After=network.target

[Service]
ExecStart=/usr/bin/python3 /srv/app/main.py
Restart=on-failure
RestartSec=5
User=deploy

[Install]
WantedBy=multi-user.target
6
ExecStart is the exact command systemd runs to start the service — no shell wrapping, so shell features like pipes need an explicit /bin/sh -c.
7
Restart=on-failure is what makes systemd bring the process back up automatically if it exits with a nonzero code or crashes.
Renders
systemctl daemon-reload && systemctl enable --now myapp.service — loads the unit, starts it, and marks it to start again on boot.

Why this works: systemd reads this unit file to know how to start (ExecStart), and under what conditions to restart (Restart=on-failure, waiting RestartSec seconds), the process — turning a plain python3 main.py into a service that survives a crash without a human noticing, which a bare foreground process cannot do on its own.

Writing a cron job that works when run manually but silently fails from cron

Wrong

bash
# crontab -e
0 2 * * * python3 /srv/app/nightly.py
# fails silently: cron's PATH often does not include the same python3
# your interactive shell finds, and errors go nowhere by default

Better

bash
# crontab -e
0 2 * * * /usr/bin/python3 /srv/app/nightly.py >> /var/log/nightly.log 2>&1
# absolute path to the interpreter, and both stdout and stderr redirected
# to a file that can actually be checked

What you see: The job runs fine typed at a prompt, but the scheduled cron run does nothing observable — no output, no error, no file written.

Why: cron runs jobs in a minimal, non-interactive shell with a much smaller PATH than a login shell, and by default any output or error a job produces is emailed (often to nowhere useful) rather than left visible. Using an absolute interpreter path removes the PATH dependency, and redirecting stdout/stderr to a real log file turns a job that fails silently into one whose failure is visible.

ssh, scp, systemd, cron — one row each

ssh, scp, systemd, cron — one row each
ToolSolvesCommon invocation
ssh"open a shell on that other machine"ssh deploy@10.0.0.5
scp"copy this file to/from that machine"scp app.tar.gz deploy@10.0.0.5:/srv/app/
systemd"keep this service running, restart it if it dies"systemctl restart myapp.service
cron"run this command every night at 2am"0 2 * * * /usr/bin/python3 /srv/app/nightly.py

Together

python
import subprocess

# Run a command on a remote host via ssh, from Python (illustrative — needs
# a real reachable host and configured key-based auth to actually connect).
result = subprocess.run(
    ["ssh", "deploy@10.0.0.5", "systemctl", "is-active", "myapp.service"],
    capture_output=True, text=True,
)
print(result.stdout.strip())

Remember: ssh opens a remote shell, scp copies files over the same connection. systemd keeps a service running and restarts it on crash (Restart=on-failure); cron runs a command on a schedule — always use an absolute interpreter path and redirect output to a log file.

See also: process inspection toolset · scheduling and cron

Advertisement