beginnerintermediateadvanced

Python · Data Serialization

Data Serialization

JSON, pickle and its security risks, CSV, YAML, the binary formats outside the stdlib, and how a serialized format evolves without breaking — verified against real executed code and official docs

dumps / loadsjson, pickle, and yaml all round-trip the same way
Never unpickle untrusted dataloads can execute arbitrary code via __reduce__
CSV quotinga naive split(",") breaks on a field containing a comma
Schema evolutionadd optional fields with defaults; never remove or rename a live one

1.JSON

json.dumps(obj) turns a Python dict, list, str, int, float, bool, or None into JSON text. json.loads(text) turns JSON text back into those same Python types.

Key features

json.dumps(obj) returns a str; json.dump(obj, file) writes straight to a file object
json.loads(text) returns a str; json.load(file) reads straight from a file object
Python True/False/None become JSON true/false/null, and back, automatically
dumps(obj, indent=2) pretty-prints; without indent, output is one compact line
A type dumps cannot handle (datetime, set, a custom object) raises TypeError, not a silent conversion

json — the four functions and the type mapping

json — the four functions and the type mapping
Call	Direction	Notes
json.dumps(obj)	Python -> JSON str	add indent=2 for pretty output, sort_keys=True for stable ordering
json.dump(obj, fp)	Python -> JSON, written to a file object	no return value; fp must be opened in text mode
json.loads(text)	JSON str -> Python	raises json.JSONDecodeError on malformed input
json.load(fp)	JSON in a file object -> Python	fp must be opened in text mode

dumps and loads are inverses

Python dict

{"user_id": 4821, ...}

json.dumps()

Python object -> JSON text

JSON text

a plain str, ready to write or send

Command

python

import json

text = json.dumps(obj, indent=2)   # Python -> JSON string
obj = json.loads(text)             # JSON string -> Python

json.dumps/loads convert between JSON text and dict/list/str/int/float/bool/None only — anything else needs default=str or a manual conversion first.

2.Pickle — including security risks

pickle.dumps(obj) serializes almost any Python object — including sets, custom classes, and datetimes — into bytes. pickle.loads(data) restores it. Unlike JSON, loading pickle data can run arbitrary code, so you only ever unpickle data you created or explicitly trust.

Key features

pickle.dumps(obj) serializes nearly any Python object — sets, custom classes, datetimes, functions
pickle.loads(data) restores it, but by construction can call arbitrary functions during loading
The official docs state plainly: "the pickle module is not secure. Only unpickle data you trust."
A class controls what runs on load through __reduce__ — pickle calls whatever it returns
json is the safer default for untrusted or cross-language data; pickle is for trusted, Python-only data

pickle — the four functions

pickle — the four functions
Call	Direction	Notes
pickle.dumps(obj)	Python -> bytes	protocol=pickle.HIGHEST_PROTOCOL for the most compact, fastest format
pickle.dump(obj, fp)	Python -> bytes, written to a file object	fp must be opened in binary mode ("wb")
pickle.loads(data)	bytes -> Python	never call on data from an untrusted source
pickle.load(fp)	bytes in a file object -> Python	fp must be opened in binary mode ("rb")

loads can execute code, not just read data

Untrusted bytes

from a network request, upload, or cache

pickle.loads()

runs whatever __reduce__ tells it to

Arbitrary code executes

in your process, with your permissions

Command

python

import pickle

blob = pickle.dumps(obj)   # Python object -> bytes
obj = pickle.loads(blob)   # bytes -> Python object (trusted source only)

pickle.loads can execute arbitrary code via __reduce__ — only unpickle data you created or explicitly trust; use json for anything untrusted or cross-language.

3.Binary serialization formats: MessagePack, Protocol Buffers, Avro

MessagePack, Protocol Buffers, and Avro all encode data as compact binary instead of text like JSON. MessagePack is a drop-in binary JSON with no schema; Protocol Buffers and Avro both use a schema, which is what makes safe format evolution possible.

Key features

None of the three are in the Python stdlib — each needs a separate package (msgpack, protobuf, fastavro or similar)
MessagePack has no schema — it is a binary drop-in for JSON, same data model, smaller and faster to parse
Protocol Buffers requires a compiled .proto schema; fields are matched by numbered tags, not names, in the wire format
Avro requires a schema too, but resolves reader/writer schema differences by field name, which the project itself cites as its advantage over Protobuf/Thrift for evolution
Pick by what you need: no-schema flexibility (MessagePack), typed cross-language RPC (Protocol Buffers), or a format built around schema evolution and Hadoop/Kafka-style pipelines (Avro)

MessagePack vs. Protocol Buffers vs. Avro

MessagePack vs. Protocol Buffers vs. Avro
Format	Schema	Typical use	Source
MessagePack	None — same dynamic model as JSON	Fast, compact drop-in replacement for JSON (caches, queues)	msgpack.org
Protocol Buffers	Required (.proto, compiled)	Typed cross-language RPC — gRPC service contracts	protobuf.dev
Avro	Required (JSON-defined schema)	Schema evolution in data pipelines — Kafka, Hadoop	avro.apache.org

Command

text

MessagePack   — no schema,        binary JSON             — msgpack.org
Protocol Buffers — schema (.proto), numbered fields, typed  — protobuf.dev
Avro          — schema (JSON),    fields resolved by name  — avro.apache.org

MessagePack = schema-free binary JSON; Protocol Buffers = typed schema with numbered fields for RPC; Avro = schema-based format built around resolving reader/writer differences by name.

4.CSV

The csv module reads and writes comma-separated rows. csv.DictReader/DictWriter work with dicts keyed by column name; csv.reader/writer work with plain lists — both handle quoting a field that itself contains a comma, which splitting a line on "," yourself gets wrong.

Key features

csv.DictWriter(file, fieldnames=[...]) writes rows from dicts; writeheader() writes the column names first
csv.DictReader(file) reads rows as dicts, using the first row as the column names automatically
csv.writer(file) / csv.reader(file) work with plain lists instead of dicts
A field containing a comma is automatically quoted on write, and correctly un-quoted on read
Every value read back is a str — csv has no concept of int/float/bool; convert columns manually

Command

python

import csv

writer = csv.DictWriter(file, fieldnames=["user_id", "name"])
writer.writeheader()
writer.writerow({"user_id": 101, "name": "Ada Lovelace"})

reader = csv.DictReader(file)
rows = list(reader)   # [{"user_id": "101", "name": "Ada Lovelace"}, ...]

csv.DictReader/DictWriter handle comma-in-field quoting correctly; every value read back is a str, so numeric columns need explicit conversion.

5.YAML

YAML is a human-readable data format that uses indentation instead of braces — most config files (Docker Compose, GitHub Actions, Kubernetes) are YAML. The third-party PyYAML package reads it with yaml.safe_load(text) and writes it with yaml.safe_dump(obj).

Key features

PyYAML is not in the stdlib — pip install pyyaml, then import yaml
yaml.safe_load(text) parses YAML text into Python dict/list/str/int/float/bool/None
yaml.safe_dump(obj) serializes those same types back into YAML text
Always use safe_load, never plain yaml.load — load can construct arbitrary Python objects from untrusted input
Indentation defines nesting; a YAML list item starts with "- "

Command

yaml

service: billing-api
replicas: 3
env:
  LOG_LEVEL: info
  TIMEOUT_MS: 3000
regions:
  - us-east-1
  - eu-west-1

yaml.safe_load/safe_dump for the JSON-like round trip; never plain yaml.load on untrusted input — it can construct arbitrary Python objects via type tags.

6.Serialization/deserialization costs

Every format spends CPU time encoding and decoding, and bytes on the wire or on disk — text formats like JSON and YAML are readable but larger and slower to parse; binary formats like pickle, MessagePack, Protobuf, and Avro are smaller and faster but not human-readable.

Key features

Encoding (serialize) and decoding (deserialize) both cost CPU time — this cost is paid on every call, not once
Text formats (JSON, YAML, CSV) are larger on the wire and slower to parse than schema-based binary formats
A schema-based binary format (Protobuf, Avro) skips re-encoding field names into every message, cutting size further
For a hot path (called per-request, at high volume), format choice is a real, measurable cost — for a one-off config file, it is not
Only trust a "format X is faster" claim backed by a benchmark on your actual payload shape — general claims vary by data shape and library version

Command

python

import json, pickle

payload = {"user_id": 4821, "name": "Priya Shah", "active": True, "roles": ["admin", "editor"]}
len(json.dumps(payload).encode("utf-8"))   # text format: readable, larger
len(pickle.dumps(payload))                 # binary format: not readable, comparable or smaller

Serialization cost is real CPU + bytes spent per call — worth measuring on a hot path with your actual payload, not assumed from a general "format X is faster" claim.

7.Schema evolution and backward compatibility

Schema evolution is changing what a serialized record looks like — adding, removing, or renaming a field — without breaking readers still using the old schema, or old data still on disk written with it. Backward compatibility means a new reader can still read old data.

Key features

Adding a new OPTIONAL field with a default is safe — old data simply gets the default when read by the new schema
Removing a field a reader still expects, or renaming one, is the classic breaking change
Backward compatible: a NEW reader can read OLD data. Forward compatible: an OLD reader can read NEW data
Avro resolves reader/writer schema differences by matching FIELD NAMES, which the project's own docs cite as its advantage over Protobuf/Thrift's numbered-tag approach for evolution
Protocol Buffers achieves similar safety differently — every field has a permanent numbered tag; reusing a retired tag number for a new field is the classic Protobuf compatibility break

Safe vs. breaking schema changes

Safe vs. breaking schema changes
Change	Safe?	Why
Add a new optional field with a default	Safe	old data simply gets the default value when read by the new schema
Remove a field nothing else still reads	Usually safe	readers that never used it are unaffected
Remove a field an existing reader expects	Breaking	that reader either errors or silently loses data it needed
Rename a field	Breaking (looks like remove + add)	a name-based resolver (Avro) sees the old name gone and a new one appear
Change a field's type (e.g. str -> int)	Breaking	old data in the old type does not parse cleanly as the new type
Reuse a retired Protobuf field-tag number	Breaking	old data tagged with that number is now misread as the new field

Command

text

Safe:     add optional field with a default
Safe:     remove a field nothing still reads
Breaking: remove a field an existing reader expects
Breaking: rename a field
Breaking: change a field's type

Add optional fields with defaults; never remove or rename a field a live reader still expects — that is what breaks backward compatibility, not adding to a schema.

Spotted a mistake on this page? Tell us.

Read Sheet