beginnerintermediateadvanced

Python · CLI and Automation

CLI and Automation

Structuring a command-line tool, and the patterns unattended scripts need to be safe to run again

Subcommandsadd_subparsers() splits one CLI into tool migrate / tool status
Exit codes0 means success — that is the one thing automation actually reads
check=Truea failed subprocess step stops the script instead of continuing silently
Cron-safelog instead of print, and be safe to run twice

1.CLI application structure

add_subparsers() splits one program into named subcommands (tool migrate, tool status), each with its own arguments and its own function — instead of one parser with a growing pile of flags.

Key features

parser.add_subparsers(dest="command", required=True) turns "tool X" into a named subcommand X
sub.add_parser("migrate") returns its own ArgumentParser — arguments added to it apply only to that subcommand
set_defaults(func=handler) attaches a handler function directly to a subcommand, avoiding an if/elif dispatch chain
add_argument_group("connection") clusters related flags together in --help output without changing parsing
A real CLI keeps argument parsing in one function and business logic in another, so the logic is testable without argv

Structuring a multi-command CLI

Structuring a multi-command CLI
Call	Effect
parser.add_subparsers(dest="command")	enables tool <command> ... syntax; args.command holds which one ran
sub.add_parser("migrate")	a full ArgumentParser scoped to the migrate subcommand only
migrate_p.set_defaults(func=cmd_migrate)	args.func is cmd_migrate after parsing — call it directly, no dispatch chain
parser.add_argument_group("connection")	groups related flags under a labeled heading in --help
required=True on add_subparsers	parse_args() errors if no subcommand is given, instead of silently doing nothing

One CLI, two subcommands, two handlers

command =="migrate"command =="status"

sys.argv

dbtool migrate --target 0007

ArgumentParser

add_subparsers(dest="command")

migrate subparser

--target flag, func=cmd_migrate

status subparser

func=cmd_status

args.func(args)

runs the matched handler

Command

python

parser = argparse.ArgumentParser(prog="tool")
sub = parser.add_subparsers(dest="command", required=True)

migrate_p = sub.add_parser("migrate")
migrate_p.add_argument("--target", default="latest")
migrate_p.set_defaults(func=cmd_migrate)

args = parser.parse_args()
args.func(args)   # dispatches without an if/elif chain

add_subparsers() turns one flat argument list into named subcommands, each with its own arguments and its own set_defaults(func=...) handler — no if/elif dispatch needed.

2.Environment variables and exit codes

os.environ.get("KEY", default) reads configuration without crashing on a missing key; sys.exit(code) reports success (0) or failure (nonzero) to whatever called the script.

Key features

os.environ.get("KEY", default) never raises; os.environ["KEY"] raises KeyError if unset
Exit code 0 means success by convention; any nonzero code means failure
An unhandled exception exits with code 1 automatically — sys.exit(1) makes that explicit and intentional
A shell reads $? (or %errorlevel% on Windows) right after a command to check its exit code
Distinct nonzero codes (1 = general error, 2 = usage error) let a caller distinguish failure reasons without parsing text

Command

python

import os, sys

api_key = os.environ.get("API_KEY")
if api_key is None:
    print("API_KEY is not set", file=sys.stderr)
    sys.exit(1)

sys.exit(0)   # success

os.environ.get("KEY", default) reads config without crashing on a missing key; sys.exit(0) means success, any nonzero code means failure — that is what automation actually checks, not the printed text.

3.Subprocesses and shell integration

subprocess.run(cmd, check=True) runs an external command and raises CalledProcessError on a nonzero exit, instead of leaving a script to silently continue after a failed step — the pattern real automation scripts use to fail loudly.

Key features

subprocess.run(cmd, check=True) raises subprocess.CalledProcessError if the command exits nonzero
Without check=True, a failed command's nonzero returncode is silently available but never inspected automatically
Piping one process into another in Python means passing one's captured stdout as the next Python step's input — not a shell |
shell=True runs the command through the system shell, enabling pipes/globbing/env expansion — and shell-injection risk with untrusted input
Prefer a list of arguments and shell=False (the default) unless a specific shell feature is actually needed

subprocess patterns for automation scripts

subprocess patterns for automation scripts
Call	Effect
subprocess.run(cmd, check=True)	raises CalledProcessError if the command exits nonzero — the script cannot silently continue
except subprocess.CalledProcessError as e	catches a failed command; e.returncode holds its exit code
subprocess.run(cmd, capture_output=True, text=True).stdout	captures the command's output as a string for the script to process further
subprocess.run("cmd1 \| cmd2", shell=True)	runs a real shell pipeline — only when the shell feature itself is required
subprocess.run([sys.executable, "-c", code])	runs another Python process — useful for isolating a step's environment or crash

A script chaining two subprocess steps, failing loudly on step 2

exit code 0exit code!= 0

step 1

subprocess.run(..., capture_output=True)

captured stdout

read into Python as a string

step 2

subprocess.run(cmd2, check=True)

script continues

CalledProcessError raised

script stops, nonzero exit propagates

Command

python

import subprocess

try:
    subprocess.run(["some-tool", "--flag"], check=True)
except subprocess.CalledProcessError as e:
    print("failed with code", e.returncode)

subprocess.run(cmd, check=True) raises on a failed command instead of silently continuing; pass a list of args (not a shell string) unless shell=True is a deliberate, justified choice.

4.Scheduling and cron

cron runs a script on a schedule from outside the process; a script invoked by cron must be idempotent and exit nonzero on failure. A long-running process instead schedules its own work in-process, typically with APScheduler.

Key features

A script invoked by cron gets no interactive terminal — print() output is lost unless redirected or logged to a file
A cron job should be idempotent — running it twice in a row must not double-apply the same work
The script must exit nonzero on failure — cron uses the exit code to decide whether to email a failure notice
A long-running process (a web server, a worker) schedules its own recurring work with a library like APScheduler instead of relying on cron to restart it
APScheduler's BackgroundScheduler runs jobs on a schedule inside the same process, using a cron-like trigger string or an interval

Command

python

# cron-invoked script: log, don't print; exit nonzero on failure
import logging, sys

logging.basicConfig(filename="/var/log/cleanup.log", level=logging.INFO)

try:
    run_cleanup()
except Exception:
    logging.exception("cleanup failed")
    sys.exit(1)
sys.exit(0)

A cron-invoked script must log (not print), be safe to re-run, and exit nonzero on failure — cron's failure detection reads only the exit code. A long-running process schedules its own work in-process instead, typically with APScheduler.

5.Automation scripts and maintenance scripts

An automation or maintenance script (cleanup, log rotation, cache eviction) is designed to run unattended and repeatedly — it logs what it did, is safe to re-run, and reports a count instead of relying on a human watching it.

Key features

A maintenance script should be idempotent — safe to run again immediately after a partial or failed run
Logging (not print) is what a human reviews after the fact, since nobody watches an unattended run live
A summary count (processed=N, errors=N) at the end is the fastest way to confirm a run actually did something
A dry-run mode (--dry-run) that logs the actions it WOULD take, without taking them, is standard for anything destructive
Scope a destructive action (delete, evict) as narrowly and explicitly as possible — a broad glob is how a cleanup script deletes the wrong thing

Command

python

import logging

logger = logging.getLogger("cleanup")
processed = errors = 0
for item in items_to_check():
    if is_stale(item):
        remove(item)
        processed += 1
        logger.info("removed %s", item)
logger.info("done: processed=%d errors=%d", processed, errors)

A maintenance script logs each action, tracks a processed/errors summary, supports --dry-run for anything destructive, and is safe to run twice — write for nobody watching, not for a live terminal.

6.Data migrations

A data migration script transforms existing records from an old shape to a new one — read, transform, write — and should support --dry-run and a way to verify results before the change is irreversible.

Key features

A data migration reads records, transforms each one, and writes the result — the same read/transform/write shape regardless of the data source
Run it in --dry-run first: log the transformation each record WOULD get, without writing anything
Process in batches (see batch jobs) rather than loading an entire table into memory at once
Keep a way to verify after writing — a count comparison, a spot check — since the "before" state may not be recoverable
A schema-level migration (adding a column) is usually handled by the ORM's own migration tool, not hand-written Python — see that framework's migrations concept

Command

python

def migrate_record(record):
    return {**record, "status": record["status"].lower()}   # transform

for batch in chunked(fetch_records(), size=500):
    transformed = [migrate_record(r) for r in batch]
    if not dry_run:
        write_batch(transformed)
    logger.info("migrated batch of %d", len(batch))

A data migration is read/transform/write over existing records — run it with --dry-run first, process in batches, and keep a way to verify the result, since the pre-migration data may not come back.

7.Batch jobs

A batch job processes a large collection in fixed-size chunks (e.g. 500 records at a time) instead of one at a time or all in memory at once — bounding both memory use and the size of a single failure.

Key features

A generator that yields fixed-size chunks keeps memory bounded regardless of how large the full input is
Each batch should be independently retriable — a failure in batch 12 should not force redoing batches 1 through 11
Batch size is a tuning knob: too small adds per-batch overhead, too large loses the memory/failure-isolation benefit
Log progress per batch (batch N of M, or a running total) so a long job is observable while it runs
A batch job and a data migration are often the same script — chunking is the mechanism, migrating is one use of it

Command

python

def chunked(iterable, size):
    chunk = []
    for item in iterable:
        chunk.append(item)
        if len(chunk) == size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

for batch in chunked(records, size=500):
    process_batch(batch)

A batch job processes fixed-size chunks — bounded memory, and a failure loses at most one chunk's worth of progress instead of the whole run.

8.Deployment tooling and developer tooling

Deployment and developer-tooling scripts wrap other command-line tools (a build, a linter, git) with subprocess, and gate their own next step on that tool's exit code — not by parsing its printed output.

Key features

A deployment script chains subprocess calls (build, then push, then deploy), stopping at the first failure via check=True
A developer-tooling script (a pre-commit check, a project scaffolder) automates a repetitive local task the same way a CI script automates a remote one
Gate on the exit code (returncode / CalledProcessError), not on parsing the tool's stdout — output format is not a stable contract, the exit code is
sys.version_info or a subprocess call to the target tool's --version confirms a required tool is present and compatible before proceeding
A deployment script should log which step is running before it runs — the point it stopped at is the first thing anyone debugging it will need

Command

python

steps = [
    ["python", "-m", "build"],
    ["twine", "check", "dist/*"],
    ["twine", "upload", "dist/*"],
]
for step in steps:
    logger.info("running: %s", " ".join(step))
    subprocess.run(step, check=True)   # stops at the first failing step

A deployment or dev-tooling script chains subprocess calls and gates each next step on the previous one's exit code (check=True or returncode == 0) — never on scraping printed text, which is not a stable contract.

Spotted a mistake on this page? Tell us.

Read Sheet