beginnerintermediateadvanced

Python · File and Resource Handling

File and Resource Handling

Reading and writing files correctly at any size, cleaning up scratch files and open handles automatically, and bundling/compressing data — verified with real temp files and archives

Text vs. binary mode"r" decodes with an encoding; "rb" returns raw bytes, untouched
Stream, don't loadfor line in f: keeps memory flat regardless of file size
with open(...) as f:guarantees close() on every exit path, including an exception
gzip / zipfile / tarfilecompress a stream, or bundle multiple files into one archive

1.File modes, binary vs. text files, and encoding

open(path, "r") reads text and decodes bytes to str using an encoding — always pass encoding="utf-8" explicitly. open(path, "rb") reads raw bytes with no decoding at all — no encoding applies.

Key features

open(path, "r") — the default mode — returns str; open(path, "rb") returns bytes, with no decoding
Text mode always needs an encoding to decode bytes into str — always pass encoding="utf-8" explicitly rather than relying on the platform default
A file that is not actually text (an image, a zip, a compiled binary) must be opened "rb" — opening it "r" either raises UnicodeDecodeError or silently corrupts it
A UnicodeDecodeError means the bytes on disk do not match the encoding you told Python to assume — check the real encoding, do not just retry a different one at random
Path.read_text(encoding=...) / Path.read_bytes() are pathlib shortcuts for open().read() that also close the file automatically

Common mode strings

Common mode strings
Mode	Meaning	Returns
"r"	Read, text (default)	str
"rb"	Read, binary	bytes
"w"	Write, text — truncates existing content	str in, nothing back
"a"	Append, text — writes are added to the end	str in, nothing back
"x"	Exclusive create — raises FileExistsError if the path already exists	str in, nothing back

Text mode vs. binary mode

Text mode ("r"/"w")

Decodes bytes to str on read, encodes str to bytes on write
Needs an encoding — always pass encoding="utf-8" explicitly
For files that ARE genuinely text: source code, JSON, CSV, logs

Binary mode ("rb"/"wb")

No decoding at all — you get and give raw bytes
No encoding parameter — bytes have no encoding
For files that are NOT text: images, zips, compiled binaries

Command

python

with open(path, "r", encoding="utf-8") as f:   # text -- returns str
    text = f.read()

with open(path, "rb") as f:                     # binary -- returns bytes
    data = f.read()

from pathlib import Path
Path(path).read_text(encoding="utf-8")          # shortcut for the "r" case
Path(path).read_bytes()                         # shortcut for the "rb" case

The mode string decides str ("r") vs. bytes ("rb") — text mode always needs an explicit encoding (encoding="utf-8"), binary mode never takes one. A file that is not actually text must open "rb".

2.Streaming files and large-file processing

for line in open(path): reads one line at a time instead of loading the whole file into memory — essential once a file is bigger than the memory available to hold it. f.read(size) does the same thing in fixed-size chunks for files with no line structure.

Key features

for line in open(path): reads one line at a time — memory use stays constant regardless of total file size
f.read(size) reads at most size bytes per call — the pattern for binary or line-less data
while chunk := f.read(size): (the walrus operator) is the idiomatic loop — it stops automatically when read() returns an empty bytes/str
f.read() with no argument (or Path.read_text()/read_bytes()) loads the ENTIRE file into memory — correct for small files, a real risk for large ones
A generator function that yields chunks keeps a large-file processing pipeline's memory use flat even when downstream code processes millions of records

Loading everything vs. streaming, by file shape

Loading everything vs. streaming, by file shape
File shape	Whole-file approach	Streaming approach
Line-structured text (logs, CSV)	f.read().splitlines()	for line in f:
No line structure (binary, a single JSON blob)	f.read()	while chunk := f.read(8192):
Very large, still line-structured	list(f) — all lines held at once	for line in f: process(line) — one line held at a time

One chunk in memory at a time, however large the file

next chunkempty read

big.txt on disk

size irrelevant — never fully loaded

f.read(size)

returns at most `size` bytes

yield chunk

hands back ONE chunk, pauses here

process(chunk)

downstream code, one piece at a time

b'' — EOF

loop stops automatically

Command

python

# Line-structured text -- constant memory regardless of file size
with open(path) as f:
    for line in f:
        process(line)

# No line structure -- fixed-size chunks
def read_in_chunks(path, size=8192):
    with open(path, "rb") as f:
        while chunk := f.read(size):
            yield chunk

for line in f: and while chunk := f.read(size): both process a file in fixed, small pieces — memory use stays flat regardless of total file size. f.read() with no argument loads everything at once.

3.Temporary files

tempfile.TemporaryDirectory() creates a scratch directory in a with block and deletes it (and everything in it) automatically on exit. tempfile.NamedTemporaryFile() does the same for a single file.

Key features

tempfile.TemporaryDirectory() as a context manager deletes the directory and everything inside it when the with block exits
tempfile.NamedTemporaryFile(delete=True) (the default) deletes the file as soon as it is closed — delete=False keeps it, for handing the path to another process
tempfile.mkstemp()/mkdtemp() create a temp file/dir WITHOUT auto-cleanup — the caller is responsible for removing it
Temp files are created in a secure, OS-chosen location (tempfile.gettempdir()) with a randomized name, which avoids collisions and predictable-filename attacks
On Windows, a file opened by NamedTemporaryFile cannot always be reopened by name while still held open elsewhere — delete=False plus closing it first is the portable pattern for handing the path to another process

The tempfile toolkit

The tempfile toolkit
Function	Creates	Auto-cleanup
TemporaryDirectory()	a directory	yes, on with-block exit
NamedTemporaryFile()	a file with a real path	yes, on close (delete=True default)
NamedTemporaryFile(delete=False)	a file with a real path	no — caller must os.remove() it
mkstemp() / mkdtemp()	a file / a directory	no — lower-level, manual cleanup

Command

python

import tempfile

with tempfile.TemporaryDirectory() as td:
    ...                                    # td is deleted, with contents, on exit

with tempfile.NamedTemporaryFile(mode="w", delete=True) as tf:
    tf.write("data")
    tf.flush()
    ...                                    # tf's file is deleted on close

TemporaryDirectory() as a with-block context manager deletes itself and its contents automatically on exit. NamedTemporaryFile(delete=False) is the portable choice when another process or a reopened handle needs the path — clean it up manually with os.remove().

4.File permissions

Path.chmod(0o644) (or os.chmod(path, 0o644)) sets a file's permission bits from Python — the same octal numbers chmod uses at the shell. stat.S_IMODE(path.stat().st_mode) reads them back.

Key features

Path(p).chmod(0o644) and os.chmod(p, 0o644) do the same thing — pick whichever fits the surrounding code (Path object vs. string path)
stat.S_IMODE(path.stat().st_mode) extracts just the permission bits from st_mode, which also encodes the file TYPE (regular file, directory, symlink)
0o644 = owner read+write, group read-only, other read-only — the common default for a file meant to be edited by its owner and read by others
0o755 is the equivalent for something that also needs to be executable (a script, a directory that needs to be entered)
On Windows, chmod only reliably controls the read-only bit — the full owner/group/other POSIX model is not honored the same way, a real, verified difference worth knowing rather than assuming portability

Common octal permission values

Common octal permission values
Octal	Meaning	Typical use
0o644	owner rw, group r, other r	a regular file others may read
0o600	owner rw, group —, other —	a private file (secrets, credentials)
0o755	owner rwx, group rx, other rx	an executable script, or a directory
0o444	owner r, group r, other r	a read-only file — no one can write without changing the mode first

Command

python

from pathlib import Path
import stat

p = Path("file.txt")
p.chmod(0o644)                              # set permissions
mode = stat.S_IMODE(p.stat().st_mode)       # read them back
print(oct(mode))                            # '0o644'

Path.chmod(0o644) / os.chmod(p, 0o644) set permission bits; stat.S_IMODE(p.stat().st_mode) reads them back, masking off the file-type bits st_mode also carries. On Windows, chmod only reliably controls the read-only bit — it is not the full POSIX model.

5.Compression and archives

gzip compresses a single stream of bytes. zipfile and tarfile bundle multiple files (and directories) into one archive — tarfile can also apply gzip compression to the whole bundle at once ("w:gz" mode).

Key features

gzip.open(path, "wb") writes compressed bytes; gzip.open(path, "rt", encoding=...) reads them back as decompressed text directly
zipfile.ZipFile(path, "w").write(src, arcname=...) adds one file to a zip under a chosen name inside the archive
tarfile.open(path, "w:gz") creates a gzip-compressed tar archive in one step — "w" alone creates an uncompressed tar
ZipFile.namelist() / TarFile.getnames() list an archive's contents without extracting anything
Extracting an archive from an untrusted source needs care — a maliciously crafted archive can contain paths like ../../etc/passwd (path traversal); Python 3.12+'s tarfile defaults to a safer extraction filter for exactly this reason

gzip, zipfile, and tarfile — one file each, one row each

gzip, zipfile, and tarfile — one file each, one row each
Module	Bundles multiple files?	Compresses?
gzip	no — one stream in, one out	yes
zipfile	yes	yes (DEFLATE, per-file)
tarfile ("w")	yes	no
tarfile ("w:gz")	yes	yes (gzip, whole archive)

Command

python

import gzip, zipfile, tarfile

with gzip.open("out.gz", "wb") as f:        # single-stream compression
    f.write(data)

with zipfile.ZipFile("bundle.zip", "w") as zf:
    zf.write("notes.txt", arcname="notes.txt")

with tarfile.open("bundle.tar.gz", "w:gz") as tf:
    tf.add("notes.txt")

gzip compresses a single stream; zipfile and tarfile bundle multiple files, and tarfile's "w:gz" mode does both in one step. Never extractall() an untrusted archive without a path-traversal-safe filter (filter="data" on Python 3.12+).

6.Resource cleanup

with open(path) as f: guarantees f.close() runs even if an exception happens inside the block — a plain f = open(path) followed by f.close() at the end does not, if something raises before that line runs.

Key features

with open(path) as f: calls f.close() automatically when the block exits, for any reason — normal completion, a return, or an exception
A file object left open (no with, no explicit close()) is NOT guaranteed to flush its write buffer to disk — data can be lost, not just "left open"
The OS enforces a limit on how many file descriptors one process can have open simultaneously — leaking file handles in a long-running process eventually raises OSError: Too many open files
A custom object supporting the same protocol (__enter__/__exit__) gets the same automatic-cleanup guarantee as a built-in file object inside a with block
__exit__ runs even when the code inside the with block raises — that guarantee, not just convenience, is with\'s actual purpose

Cleanup guarantee, with vs. without with

Cleanup guarantee, with vs. without with
Pattern	Runs close() on exception?	Runs close() on early return?
with open(path) as f: ...	yes, always	yes, always
f = open(path); ...; f.close()	no — an exception before f.close() skips it	no — an early return before f.close() skips it
try: ...\nfinally: f.close()	yes	yes

What with guarantees around the block it wraps

runs firstsuccess, return,OR exception

__enter__

opens/acquires the resource

the with block

normal code, may raise

__exit__

ALWAYS runs — close(), release, etc.

Command

python

with open(path) as f:
    data = f.read()
# f.close() has already run here, guaranteed -- even if the block raised

class ManagedResource:
    def __enter__(self):
        self.handle = acquire()
        return self.handle
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.handle.release()   # runs even if the with block raised
        return False             # don't suppress the exception

with open(path) as f: guarantees close() runs on every exit path — normal completion, an early return, or an exception — which a manually placed f.close() at the end of a function does not.

Spotted a mistake on this page? Tell us.

Read Sheet