Filter concepts by levelShowing all levels.

Python · Section 16

Dependency and Environment Management

Level
intermediate
Read
120 min
Concepts
8

Virtual environments and Python version management, development vs. production dependencies, environment variables and .env files, secrets, lock files and reproducible environments, and the tool landscape — pip, uv, Poetry, pip-tools, Conda, and pyenv.

What is true here

  1. A venv isolates packages, inheriting whatever Python ran it; a version manager (uv, pyenv) isolates the interpreter itself.
  2. Production dependencies are what the running app needs; development dependencies should never ship to production.
  3. A .env file needs python-dotenv's load_dotenv() to actually load — nothing built into Python reads it automatically.
  4. A secret must never appear in source control — read it from an environment variable, and rotate any secret ever committed.
  5. A lock file records the exact resolved version and hash of every dependency, making an install reproducible everywhere.

What you will be able to do

  • Explain why creating a venv does not change which Python interpreter version is available
  • Separate development and production dependencies using [project.optional-dependencies]
  • Load configuration from a .env file with python-dotenv, and explain why it must run before reading the variables
  • Handle a secret correctly — read from the environment, fail loudly if missing, never commit it
  • Explain what a lock file adds over a version range, and why it should be committed for an application
  • Compare pip and uv, and know when Poetry, pip-tools, Conda, or pyenv are the more relevant choice

Environments and dependencies

Isolating packages and Python versions, separating development from production dependencies, and configuring an application through environment variables and secrets.

Virtual environments and Python version management

coreintermediate

A virtual environment isolates a project's installed packages. A Python version manager goes one level further, isolating the interpreter itself, so one project can use 3.10 while another uses 3.13 on the same machine.

Think of it as

A venv is a private toolbox for one project's packages. A version manager is having several different-sized wrenches available at all — the venv picks which toolbox, the version manager picks which wrench even exists to put in it.

python
# venv: isolates PACKAGES for one Python version
python -m venv .venv

# version management: isolates the PYTHON VERSION itself
uv python install 3.12
uv venv --python 3.12

What we're doing: Use uv to create a venv pinned to a specific Python version, showing environment and interpreter isolation working together.

uv_venv_version.shpython
uv venv
# Using CPython 3.11.15
# Creating virtual environment at: .venv
# Activate with: .venv\Scripts\activate
1
uv venv both selects a Python interpreter version AND creates a package-isolated environment for it, in one step.
Output
Using CPython 3.11.15
Creating virtual environment at: .venv
Activate with: .venv\Scripts\activate

Why this works: uv reports exactly which interpreter it picked (3.11.15 here, from what was available on this machine) before creating the environment — making explicit the two separate concerns a venv alone does not cover: which Python, and which packages.

Assuming a venv changes the Python version available

Wrong

python
# system only has Python 3.14 installed
python -m venv .venv
.venv\Scripts\activate
python --version   # still 3.14 -- venv did NOT install 3.10

Better

python
uv python install 3.10   # actually fetches the 3.10 interpreter
uv venv --python 3.10    # NOW creates a venv using it
.venv\Scripts\activate
python --version         # 3.10

What you see: A project declaring requires-python = ">=3.10,<3.11" still runs on 3.14 inside its venv, silently, because nothing actually enforces the version at venv-creation time without a version manager.

Why: python -m venv .venv always uses the SAME interpreter that ran the command — it copies references to that interpreter, never downloads a different one. Getting an actually different Python version requires a version manager (uv, pyenv) that can install one.

Two separate concerns, two separate tools

Version manager

uv python install 3.12

fetches an interpreter build

Python 3.10, 3.12, 3.14

multiple versions, side by side

Virtual environment

uv venv --python 3.12

inherits the chosen interpreter

.venv/

isolated packages, this project only

  • Version manager — which interpreter exists at all
    • uv python install 3.12 — fetches an interpreter build
    • Python 3.10, 3.12, 3.14 — multiple versions, side by side
  • Virtual environment — which packages, for one project
    • uv venv --python 3.12 — inherits the chosen interpreter
    • .venv/ — isolated packages, this project only

Remember: A venv isolates packages, inheriting whatever Python ran it; a version manager isolates the interpreter itself.

See also: pip and uv · pip and venv

Development vs. production dependencies

coreintermediate

Production dependencies are what the running application needs to actually work. Development dependencies (pytest, ruff, mypy) are only needed while building and testing it — production should never need to install them.

Think of it as

Production dependencies are the ingredients that end up in the finished dish. Development dependencies are kitchen tools — a thermometer, a timer — genuinely necessary to cook well, but nobody eats them, and they have no reason to ship with the meal.

python
# pyproject.toml
[project]
dependencies = ["requests>=2.31,<3.0"]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff>=0.5"]

# production install:
pip install .
# development install:
pip install ".[dev]"

What we're doing: Install a package with dev extras and confirm both production and dev-only packages end up present.

dev_extras_install.shpython
pip install -e ".[dev]"
pip list | grep -E "requests|pytest"
# pytest             9.1.1
# requests           2.34.2
1
The [dev] extra pulls in the dev = [...] list from [project.optional-dependencies], on top of the base dependencies.
Output
pytest             9.1.1
requests           2.34.2

Why this works: Both appear because [dev] is additive — it never replaces the base dependencies, only adds to them. A production install (pip install . with no [dev]) would show requests but not pytest.

Shipping a Docker image built with [dev] extras installed

Wrong

python
# Dockerfile
RUN pip install ".[dev]"   # pulls in pytest, ruff, mypy -- unused at runtime

Better

python
# Dockerfile
RUN pip install .   # production dependencies only

# a SEPARATE Dockerfile stage or CI job installs [dev] for testing

What you see: The production image is noticeably larger than necessary, and carries testing/linting tools that expand its attack surface for no runtime benefit.

Why: Dev dependencies exist to support the development workflow, not the running application — installing them into a production image adds size and unnecessary packages without ever being used by the app itself.

pip install . vs. pip install ".[dev]"

pip install .

  • +Installs only [project.dependencies]
  • +requests: yes — the app imports it
  • +pytest, ruff: never installed

pip install ".[dev]"

  • −Base dependencies PLUS the dev = [...] extra
  • −requests AND pytest, ruff all present
  • −Use for local development, never production images
  • pip install .
    • Installs only [project.dependencies]
    • requests: yes — the app imports it
    • pytest, ruff: never installed
  • pip install ".[dev]"
    • Base dependencies PLUS the dev = [...] extra
    • requests AND pytest, ruff all present
    • Use for local development, never production images

Where each kind belongs

Where each kind belongs
DependencyKindNeeded in production?
requestsproductionYes — the app imports and calls it
pytestdevNo — only runs during testing
ruffdevNo — only runs during linting, never at runtime
djangoproductionYes — the app cannot run without it

Together

python
[project]
dependencies = ["requests>=2.31,<3.0"]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff>=0.5"]

Remember: Production dependencies are what the app needs; dev dependencies should never ship — [project.optional-dependencies] separates them.

See also: dependencies pinning and ranges · lock files and reproducible environments

Environment variables and .env files

corebeginner

os.environ.get("NAME") reads an environment variable set on the process. A .env file lists variables in NAME=value lines for local development — python-dotenv loads it into os.environ, since Python does not read it automatically.

Think of it as

os.environ is the shell's handwritten sticky notes stuck to the process — always there, no extra step needed. A .env file is a shopping list of notes to stick up, but nothing sticks them automatically; python-dotenv is what actually copies them onto the process.

python
# .env
DATABASE_URL=postgresql://localhost/mydb
DEBUG=true

# app.py
from dotenv import load_dotenv
load_dotenv()
import os
db_url = os.environ.get("DATABASE_URL")

What we're doing: Show a real .env file loaded with python-dotenv, confirming os.environ is empty before and populated after.

env_loading.pypython
import os
from dotenv import load_dotenv

print("before:", os.environ.get("DATABASE_URL"))
load_dotenv()
print("after:", os.environ.get("DATABASE_URL"))
print("DEBUG:", os.environ.get("DEBUG"))
4
Before load_dotenv() runs, DATABASE_URL is not in the process environment at all — nothing has read the .env file yet.
6
load_dotenv() reads .env from the current directory and copies every NAME=value line into os.environ.
Output
before: None
after: postgresql://localhost/mydb
DEBUG: true

Why this works: A .env file is plain text — Python has no built-in mechanism that reads it automatically. python-dotenv's load_dotenv() is what parses the file and calls os.environ[...] = value for each line, making the values available exactly as if they had been set on the real shell.

Reading os.environ before calling load_dotenv()

Wrong

python
import os

DATABASE_URL = os.environ.get("DATABASE_URL")   # None -- .env not loaded yet

from dotenv import load_dotenv
load_dotenv()   # too late, DATABASE_URL was already read above

Better

python
from dotenv import load_dotenv
load_dotenv()   # load FIRST

import os
DATABASE_URL = os.environ.get("DATABASE_URL")   # correctly reads the .env value

What you see: A configuration value reads as None (or a default) even though the .env file clearly defines it — no error, just a silently wrong value.

Why: os.environ.get() reads whatever is in the environment at the exact moment it runs. Calling it before load_dotenv() has populated os.environ means it reads an environment that does not have the .env values yet.

.env is inert until load_dotenv() copies it in

.env file

DATABASE_URL=... on disk

load_dotenv()

parses and copies each line

os.environ

the real process environment

os.environ.get("NAME")

reads whatever is there NOW

  • .env file — DATABASE_URL=... on disk
    • leads to load_dotenv()
  • load_dotenv() — parses and copies each line
    • leads to os.environ
  • os.environ — the real process environment
    • leads to os.environ.get("NAME")
  • os.environ.get("NAME") — reads whatever is there NOW

Reading environment configuration

Reading environment configuration
CallBehavior
os.environ["NAME"]raises KeyError if NAME is not set
os.environ.get("NAME")returns None if NAME is not set
os.environ.get("NAME", "default")returns "default" if NAME is not set
load_dotenv()reads .env from the current directory (or a given path) into os.environ

Together

python
import os
from dotenv import load_dotenv

load_dotenv()   # reads .env, sets values in os.environ
db_url = os.environ.get("DATABASE_URL", "sqlite:///default.db")

Remember: os.environ.get("NAME") reads real environment variables; a .env file needs load_dotenv() to actually load.

See also: secrets management · dev vs production dependencies

Secrets

coreintermediate

A secret is a value (API key, password, key) that must never appear in source control. Environment variables are the most common way to hand one to a running app — set by the deploy platform, or locally via a gitignored .env.

Think of it as

Committing a secret to git is not like deleting a file — it is like writing something on a whiteboard and photographing it before erasing. The photo (the git history) still exists, viewable by anyone with repo access, forever, even after the line is "removed" in a later commit.

python
import os

api_key = os.environ.get("STRIPE_API_KEY")
if api_key is None:
    raise RuntimeError("STRIPE_API_KEY is not set")

What we're doing: Show application code reading a required secret and failing loudly and immediately if it is missing, rather than continuing with a broken or empty value.

require_secret.pypython
import os

def get_required_env(name):
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(f"{name} must be set")
    return value

api_key = get_required_env("STRIPE_API_KEY")
print("loaded a key of length:", len(api_key))
4
os.environ.get returns None if the secret is missing — checked explicitly rather than trusted blindly.
5
Failing immediately and loudly here beats silently continuing with a missing or empty API key, which would fail confusingly later, deep inside a real API call.
Output
RuntimeError: STRIPE_API_KEY must be set

Why this works: A missing secret is a configuration error, not a normal runtime condition — raising immediately at startup, with a clear message naming exactly which variable is missing, is far easier to diagnose than a confusing failure deep inside whatever code eventually tries to use an empty string as an API key.

Committing a .env file with real secrets to git

Wrong

python
# .gitignore does NOT list .env
git add .
git commit -m "add config"
# .env, with a real STRIPE_API_KEY inside, is now in git history forever

Better

python
# .gitignore
.env

# commit a TEMPLATE instead, with no real values:
# .env.example
# STRIPE_API_KEY=

What you see: A real API key or password is discoverable by anyone who can read the repository — including in its history, even after a later commit deletes the file.

Why: .gitignore only prevents git from tracking a NEW file — it does nothing for a file already committed. If a secret was ever committed, the fix is rotating the secret (issuing a new one) and removing the old file, not just deleting it going forward.

Where a secret should live — least to most safe

Hardcoded in a .py file

ships in every clone, forever in git history

.env file, gitignored

OK for local development only

Env var set by the deploy platform

standard for production

Dedicated secret manager

Vault, AWS Secrets Manager — rotation, access control

  1. Hardcoded in a .py file — ships in every clone, forever in git history
  2. .env file, gitignored — OK for local development only
  3. Env var set by the deploy platform — standard for production
  4. Dedicated secret manager — Vault, AWS Secrets Manager — rotation, access control

Where a secret should (and should not) live

Where a secret should (and should not) live
LocationSafe?
Hardcoded in a .py fileNo — ships in every clone, forever in git history
.env file, gitignoredOK for local development only
Environment variable set by the deploy platformYes — standard for production
A dedicated secret manager (Vault, AWS Secrets Manager)Yes — the production-grade approach, with rotation and access control

Together

python
import os

api_key = os.environ.get("STRIPE_API_KEY")
if not api_key:
    raise RuntimeError("STRIPE_API_KEY is not set")

Remember: A secret must never appear in source control — read it from the environment, fail loudly if missing, rotate if leaked.

See also: environment variables and env files

Advertisement

Reproducibility and tools

Lock files for reproducible installs, then the tool landscape — pip and uv verified hands-on, Poetry/pip-tools/Conda/pyenv covered at the depth the roadmap itself calls for ("you do not need to master every dependency manager").

Lock files and reproducible environments

coreintermediate

A lock file records the exact version — often with a hash — of every package actually installed, direct and transitive. Installing from it gives the same versions every time, on every machine, instead of whatever a range resolves to today.

Think of it as

A version range in pyproject.toml is a recipe with "a ripe tomato" as an ingredient — correct today, but which specific tomato varies by trip to the store. A lock file is the receipt from one specific trip, listing the exact tomato brand and size bought, so the next cook can buy precisely that again.

python
uv lock          # resolve and write uv.lock
uv sync           # install EXACTLY what uv.lock specifies

What we're doing: Generate a real lock file from a range-declared dependency and inspect the exact resolved version it records.

lock_demo.shpython
# pyproject.toml: dependencies = ["requests>=2.31"]
uv lock

# uv.lock (excerpt):
# [[package]]
# name = "requests"
# version = "2.34.2"
# source = { registry = "https://pypi.org/simple" }
2
uv lock resolves the range >=2.31 to one specific, exact version and writes it to uv.lock.
7
This exact version is what uv sync will install every time, on every machine, until uv lock is re-run.
Output
Resolved 6 packages in 144ms

Why this works: uv resolved requests>=2.31 to exactly 2.34.2 (the latest compatible version available at lock time) and wrote that, plus every transitive dependency (certifi, charset-normalizer, idna, urllib3) with its own exact version and file hash, into uv.lock — reproducing this exact install anywhere just means running uv sync against the same lock file.

Committing pyproject.toml but not the lock file

Wrong

python
# .gitignore
uv.lock   # excluded by mistake -- or just never committed

Better

python
# uv.lock should be committed for an application
# (libraries typically do NOT commit a lock file --
#  they need to stay compatible with a RANGE of versions)

What you see: Two developers (or a developer and CI) install slightly different transitive dependency versions from the same pyproject.toml, and a bug only reproduces on one of them.

Why: Without a committed lock file, every install re-resolves the declared ranges independently — a version range like >=2.31 can legitimately resolve to a different version tomorrow than it did today, once a new compatible release ships to PyPI.

A range resolves once, then locks to an exact version

pyproject.toml

requests>=2.31,<3.0 — a RANGE

uv lock

resolves every dep, direct + transitive

uv.lock

requests==2.34.2, exact version + hash

uv sync

installs precisely what is locked

  • pyproject.toml — requests>=2.31,<3.0 — a RANGE
    • leads to uv lock
  • uv lock — resolves every dep, direct + transitive
    • leads to uv.lock
  • uv.lock — requests==2.34.2, exact version + hash
    • leads to uv sync
  • uv sync — installs precisely what is locked

Declared range vs. locked reality

Declared range vs. locked reality
FileHolds
pyproject.tomldependencies = ["requests>=2.31,<3.0"] — a RANGE
uv.lockrequests == 2.34.2, plus every transitive dependency's exact version and hash

Together

python
# pyproject.toml
dependencies = ["requests>=2.31,<3.0"]

# uv.lock (generated)
# [[package]]
# name = "requests"
# version = "2.34.2"
# ...

Remember: A lock file records the exact resolved version and hash of every dependency — commit it for reproducible installs.

See also: dependencies pinning and ranges · pip and uv

pip and uv

coreintermediate

pip is Python's standard package installer, included with every Python install. uv is a much faster, newer tool that can install packages, create venvs, manage Python versions, and lock dependencies — often a drop-in pip replacement.

Think of it as

pip is the reliable, everywhere-available tool everyone already has. uv is a faster car built later, covering the same essential trip (install a package) plus extra stops (version management, locking) pip alone does not make.

python
# pip, the standard tool
python -m venv .venv
pip install requests

# uv, a faster all-in-one replacement
uv venv
uv pip install requests

What we're doing: Install the same package with both pip and uv in fresh environments, comparing the actual install output.

pip_vs_uv.shpython
uv venv
uv pip install requests
# Resolved 5 packages in 385ms
# Prepared 3 packages in 217ms
# Installed 5 packages in 249ms
#  + certifi==2026.7.22
#  + charset-normalizer==3.5.1
#  + idna==3.19
#  + requests==2.34.2
#  + urllib3==2.7.0
1
uv venv both picks a Python interpreter and creates the environment — one command instead of pip's separate python -m venv step.
2
uv pip install accepts the same package names and syntax pip does — the interface is designed to be a familiar drop-in.
Output
Resolved 5 packages in 385ms
Prepared 3 packages in 217ms
Installed 5 packages in 249ms
 + certifi==2026.7.22
 + charset-normalizer==3.5.1
 + idna==3.19
 + requests==2.34.2
 + urllib3==2.7.0

Why this works: uv resolved and installed the same requests + its transitive dependencies pip would, but reports timing explicitly and structures output around resolve/prepare/install phases — the same end result as pip install requests, in noticeably less wall-clock time.

pip vs. uv

pip

  • +Installed with Python by default — always available
  • +Needs the separate venv module for environments
  • +No built-in lock files — needs pip-tools/Poetry

uv

  • −Written in Rust — measurably faster resolve/install
  • −uv venv also manages Python versions directly
  • −uv lock / uv sync built in for reproducible installs
  • pip
    • Installed with Python by default — always available
    • Needs the separate venv module for environments
    • No built-in lock files — needs pip-tools/Poetry
  • uv
    • Written in Rust — measurably faster resolve/install
    • uv venv also manages Python versions directly
    • uv lock / uv sync built in for reproducible installs

Expecting uv pip install to update uv.lock

Wrong

python
uv pip install flask   # installs it, but...
ls uv.lock             # No such file or directory -- uv.lock was NOT touched

Better

python
uv add flask   # updates pyproject.toml AND uv.lock together
# or, for an already-declared dependency:
uv lock        # regenerates uv.lock from pyproject.toml
uv sync        # installs exactly what the lock file specifies

What you see: A package installed with uv pip install works locally, but disappears (or was never locked) after the next uv sync elsewhere, since it was never recorded anywhere persistent.

Why: uv pip install is the low-level, pip-compatible interface — it installs into the environment but does not touch pyproject.toml or uv.lock at all. uv add is the project-aware command that updates both together, which is what most real workflows should use instead.

pip vs. uv, by capability

pip vs. uv, by capability
Capabilitypipuv
Install a packageYesYes — faster
Create a venvOnly with the separate venv moduleBuilt in (uv venv)
Manage Python versionsNoYes (uv python install)
Lock filesNo (needs pip-tools/Poetry)Built in (uv lock)

Together

python
pip install requests
uv pip install requests   # same result, faster resolution

Remember: pip is the standard installer; uv is much faster and also handles venvs, Python versions, and lock files.

See also: pip and venv · lock files and reproducible environments

Poetry and pip-tools

standardintermediate

Poetry is an all-in-one dependency manager with its own conventions, a built-in lock file, and build/publish commands. pip-tools is lighter — it just compiles a pinned requirements.txt from a looser requirements.in.

Think of it as

Poetry is a fully furnished apartment — dependency management, locking, building, and publishing all included, in Poetry's own way. pip-tools is one extra tool added to a toolbox you already have — it only does one job (locking), leaving everything else to plain pip.

python
# Poetry
poetry init
poetry add requests
poetry install

# pip-tools
echo "requests>=2.31" > requirements.in
pip-compile requirements.in   # writes requirements.txt, fully pinned
pip install -r requirements.txt

Remember: Poetry is an all-in-one manager with its own lock file; pip-tools just compiles a pinned requirements.txt.

See also: pip and uv · lock files and reproducible environments

Conda and pyenv (when relevant)

referenceintermediate

Conda is a package/environment manager that also handles non-Python dependencies (C libraries, compiled scientific packages) — common in data science. pyenv manages multiple Python interpreter versions on one machine.

Think of it as

pip installs Python packages, assuming system libraries already exist. Conda manages the whole stack — Python plus C libraries like those numpy/scipy build on — why it shows up more in data science than typical backend work.

python
# Conda
conda create -n myenv python=3.12
conda activate myenv
conda install numpy

# pyenv (macOS/Linux)
pyenv install 3.12
pyenv global 3.12

Remember: Conda manages Python plus non-Python system dependencies; pyenv manages multiple Python interpreter versions.

See also: virtual environments and python version management · pip and uv

Advertisement