beginnerintermediateadvanced

Python · Section 44

Data Serialization

Level: intermediate
Read: 110 min
Concepts: 7

Converting Python objects to and from a storable or transmittable form — JSON and CSV as the everyday text formats, pickle for arbitrary Python objects and the real security risk of loading untrusted pickle data, YAML for human-edited config, MessagePack/Protocol Buffers/Avro as the binary formats outside the stdlib, and how serialization cost and schema evolution shape a format choice over time.

This section

What is Python Pickle? A Beginner’s Guide to Data SerializationJames Clare

What is true here

json.dumps/loads round-trip dict/list/str/int/float/bool/None only — anything else needs default=str or manual conversion.
pickle.loads can execute arbitrary code via __reduce__ — only unpickle data you created or explicitly trust; use json for anything untrusted.
csv.DictReader/DictWriter handle comma-in-field quoting a naive split(",") gets wrong.
MessagePack is schema-free binary JSON; Protocol Buffers and Avro both use a schema, which is what makes safe format evolution possible.
A schema evolves safely by adding optional fields with defaults — removing or renaming a field a live reader still expects is the breaking change.

What you will be able to do

Round-trip data through json.dumps/loads and handle a value dumps cannot serialize
Explain, with the specific __reduce__ mechanism, why pickle.loads must never run on untrusted data
Read and write CSV with csv.DictReader/DictWriter without breaking on a comma inside a field
Parse and emit YAML config with yaml.safe_load/safe_dump instead of the unsafe plain load
Choose between MessagePack, Protocol Buffers, and Avro based on whether a schema, cross-language RPC, or pipeline-style evolution is needed
Measure the actual byte-size and CPU cost of a serialization format instead of assuming one from a general claim
Evolve a serialized schema by adding optional fields with defaults, and identify a change that would break existing readers

Text and object formats

JSON as the default interchange format, pickle for arbitrary Python objects (and why loading untrusted pickle data is dangerous), CSV for tabular data, and YAML for human-edited config.

JSON

corebeginner

json.dumps(obj) turns a Python dict, list, str, int, float, bool, or None into JSON text. json.loads(text) turns JSON text back into those same Python types.

Think of it as

JSON is a text-only mailing format for a small, fixed set of Python types. dumps packs an object into that envelope; loads unpacks it on the other end — anything not in the allowed set (a datetime, a set, a custom class) has to be converted to a plain type first, or dumps raises TypeError.

python

import json

text = json.dumps(obj, indent=2)   # Python -> JSON string
obj = json.loads(text)             # JSON string -> Python

What we're doing: Round-trip a dict through json.dumps/loads, and see what happens when a value is not JSON-serializable.

json_roundtrip.pypython

import json
import datetime

data = {"user_id": 4821, "name": "Priya Shah", "active": True, "roles": ["admin", "editor"], "score": None}
encoded = json.dumps(data)
decoded = json.loads(encoded)
print(encoded)
print(decoded == data)

json.dumps({"created": datetime.datetime(2026, 8, 22)})

5: dumps converts the dict to a JSON string — True becomes true, None becomes null, in the actual text.
6: loads parses that string back into a dict with the original Python types — the round trip is lossless for these types.
10: datetime is not one of the types dumps knows how to serialize, so this line raises instead of silently stringifying it.

Output

{"user_id": 4821, "name": "Priya Shah", "active": true, "roles": ["admin", "editor"], "score": null}
True
Traceback (most recent call last):
  ...
TypeError: Object of type datetime is not JSON serializable

Why this works: json.dumps only knows how to convert the JSON-native types (dict, list, str, int, float, bool, None) — datetime is a Python-specific type with no direct JSON equivalent, so dumps raises TypeError rather than guessing a string representation for you.

Passing a non-serializable value straight to json.dumps

Wrong

python

import json, datetime

payload = {"user_id": 4821, "created": datetime.datetime.now()}
json.dumps(payload)  # TypeError: Object of type datetime is not JSON serializable

Better

python

import json, datetime

payload = {"user_id": 4821, "created": datetime.datetime.now()}
json.dumps(payload, default=str)  # falls back to str() for anything dumps doesn't know

What you see: TypeError: Object of type datetime is not JSON serializable — raised at dumps() time, not somewhere later.

Why: dumps only handles the JSON-native types directly. default=str tells it to call str() on anything else instead of raising — a quick fix for logging or debug output; a real API instead converts the value explicitly (datetime.isoformat()) so the output format is chosen deliberately, not left to str()'s default representation.

dumps and loads are inverses

Python dict

{"user_id": 4821, ...}

json.dumps()

Python object -> JSON text

JSON text

a plain str, ready to write or send

json — the four functions and the type mapping

json — the four functions and the type mapping
Call	Direction	Notes
json.dumps(obj)	Python -> JSON str	add indent=2 for pretty output, sort_keys=True for stable ordering
json.dump(obj, fp)	Python -> JSON, written to a file object	no return value; fp must be opened in text mode
json.loads(text)	JSON str -> Python	raises json.JSONDecodeError on malformed input
json.load(fp)	JSON in a file object -> Python	fp must be opened in text mode

Together

python

import json

data = {"user_id": 4821, "name": "Priya Shah", "active": True}
text = json.dumps(data, indent=2, sort_keys=True)
print(text)
print(json.loads(text) == data)

Remember: json.dumps/loads convert between JSON text and dict/list/str/int/float/bool/None only — anything else needs default=str or a manual conversion first.

Pickle — including security risks

coreintermediate

pickle.dumps(obj) serializes almost any Python object — including sets, custom classes, and datetimes — into bytes. pickle.loads(data) restores it. Unlike JSON, loading pickle data can run arbitrary code, so you only ever unpickle data you created or explicitly trust.

Think of it as

JSON is a form with fixed fields — only a few types fit. Pickle is a recipe: it can say "to rebuild this object, call this function with these arguments," and pickle.loads follows that recipe literally, including calling whatever function it names. That is exactly why loading an untrusted pickle is like running a script someone else wrote — you would not do that either.

python

import pickle

blob = pickle.dumps(obj)   # Python object -> bytes
obj = pickle.loads(blob)   # bytes -> Python object (trusted source only)

What we're doing: Round-trip an object pickle can serialize but json cannot (a set, a datetime), then show why pickle.loads can run code — a print() call standing in for anything more harmful, never executed here.

pickle_reduce.pypython

import pickle, datetime

record = {"user_id": 4821, "roles": {"admin", "editor"}, "created": datetime.datetime(2026, 8, 22, 9, 30)}
blob = pickle.dumps(record)
restored = pickle.loads(blob)
print(restored == record)


class Payload:
    def __reduce__(self):
        return (print, ("this callable runs during pickle.loads, not pickle.dumps",))


safe_demo_blob = pickle.dumps(Payload())
print("dumps succeeded, nothing printed yet")
pickle.loads(safe_demo_blob)   # the print() call executes here, on LOAD

3: record holds a set and a datetime — neither is valid JSON, but pickle handles both directly.
10: __reduce__ tells pickle exactly how to rebuild this object: call print with this one argument.
14: dumps only serializes the recipe — it does not call print(). Nothing runs yet.
15: loads follows the recipe and calls print() here. A real attack replaces print with os.system or similar — the mechanism is identical, only the callable differs.

Output

True
dumps succeeded, nothing printed yet
this callable runs during pickle.loads, not pickle.dumps

Why this works: __reduce__ lets any class tell pickle "to rebuild me, call this function with these arguments" — which is exactly how pickle restores complex objects. loads has no way to tell a legitimate reconstruction call from a malicious one; it just calls whatever __reduce__ produced. That is the entire mechanism behind why untrusted pickle data is unsafe to load.

Calling pickle.loads on data from an untrusted source

Wrong

python

import pickle

def load_session(raw_bytes_from_cookie):
    return pickle.loads(raw_bytes_from_cookie)   # attacker controls this input

Better

python

import json

def load_session(raw_text_from_cookie):
    return json.loads(raw_text_from_cookie)   # only plain data types, nothing executes

What you see: No error at all in the vulnerable version — a crafted payload runs its attacker-chosen code silently during loads(), often before any exception could be raised.

Why: The official docs state it plainly: "The pickle module is not secure. Only unpickle data you trust. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling." A cookie, upload, or network payload is not trusted by definition — json or another data-only format removes the risk entirely because loading it cannot call arbitrary functions.

loads can execute code, not just read data

Untrusted bytes

from a network request, upload, or cache

pickle.loads()

runs whatever __reduce__ tells it to

Arbitrary code executes

in your process, with your permissions

pickle — the four functions

pickle — the four functions
Call	Direction	Notes
pickle.dumps(obj)	Python -> bytes	protocol=pickle.HIGHEST_PROTOCOL for the most compact, fastest format
pickle.dump(obj, fp)	Python -> bytes, written to a file object	fp must be opened in binary mode ("wb")
pickle.loads(data)	bytes -> Python	never call on data from an untrusted source
pickle.load(fp)	bytes in a file object -> Python	fp must be opened in binary mode ("rb")

Together

python

import pickle, datetime

record = {"user_id": 4821, "roles": {"admin", "editor"}, "created": datetime.datetime(2026, 8, 22, 9, 30)}
blob = pickle.dumps(record)
restored = pickle.loads(blob)
print(restored == record)

Remember: pickle.loads can execute arbitrary code via __reduce__ — only unpickle data you created or explicitly trust; use json for anything untrusted or cross-language.

See also: json · binary serialization formats

CSV

standardbeginner

The csv module reads and writes comma-separated rows. csv.DictReader/DictWriter work with dicts keyed by column name; csv.reader/writer work with plain lists — both handle quoting a field that itself contains a comma, which splitting a line on "," yourself gets wrong.

Think of it as

CSV looks like something you could parse with text.split(","), until a field contains a comma or a newline itself — a name like "Hopper, Grace". The csv module knows the quoting rule that makes that unambiguous ("field, with comma" wrapped in quotes); split(",") does not.

python

import csv

writer = csv.DictWriter(file, fieldnames=["user_id", "name"])
writer.writeheader()
writer.writerow({"user_id": 101, "name": "Ada Lovelace"})

reader = csv.DictReader(file)
rows = list(reader)   # [{"user_id": "101", "name": "Ada Lovelace"}, ...]

What we're doing: Write rows with DictWriter, including one name containing a comma, then read them back with DictReader and confirm the comma survived correctly.

csv_roundtrip.pypython

import csv, io

buf = io.StringIO()
writer = csv.DictWriter(buf, fieldnames=["user_id", "name", "signup_date"])
writer.writeheader()
writer.writerow({"user_id": 101, "name": "Ada Lovelace", "signup_date": "2026-01-15"})
writer.writerow({"user_id": 102, "name": "Grace Hopper, PhD", "signup_date": "2026-02-03"})
print(buf.getvalue())

buf.seek(0)
reader = csv.DictReader(buf)
rows = list(reader)
print(rows)

7: "Grace Hopper, PhD" contains a comma — writer automatically wraps this field in quotes in the output text.
10: seek(0) rewinds the in-memory buffer so DictReader starts from the beginning, same as reopening a file for reading.
12: DictReader uses row 1 (the header) as keys automatically — every value comes back as a str, including user_id.

Output

user_id,name,signup_date
101,Ada Lovelace,2026-01-15
102,"Grace Hopper, PhD",2026-02-03

[{'user_id': '101', 'name': 'Ada Lovelace', 'signup_date': '2026-01-15'}, {'user_id': '102', 'name': 'Grace Hopper, PhD', 'signup_date': '2026-02-03'}]

Why this works: writer quotes "Grace Hopper, PhD" as "Grace Hopper, PhD" specifically because it contains the delimiter — without that quoting, a naive comma-split would read it as two separate fields. DictReader parses the quoting correctly and reconstructs the dict with the comma intact in one field, exactly as written.

Parsing CSV by splitting each line on ","

Wrong

python

line = '103,"Lin, Wei",2026-03-01'
fields = line.split(",")
print(fields)   # ['103', '"Lin', ' Wei"', '2026-03-01'] -- wrong, name split in two

Better

python

import csv, io

reader = csv.reader(io.StringIO('103,"Lin, Wei",2026-03-01'))
print(next(reader))   # ['103', 'Lin, Wei', '2026-03-01'] -- correct

What you see: A name like "Lin, Wei" silently splits into two fields, shifting every column after it — 'signup_date' ends up holding what should have been part of the name, with no error raised.

Why: split(",") has no concept of quoting — it splits on every comma in the line, including ones inside a quoted field. csv.reader implements the actual CSV quoting rule (a comma inside double quotes is not a delimiter), so it parses the same line correctly.

Remember: csv.DictReader/DictWriter handle comma-in-field quoting correctly; every value read back is a str, so numeric columns need explicit conversion.

YAML

standardbeginner

YAML is a human-readable data format that uses indentation instead of braces — most config files (Docker Compose, GitHub Actions, Kubernetes) are YAML. The third-party PyYAML package reads it with yaml.safe_load(text) and writes it with yaml.safe_dump(obj).

Think of it as

YAML is JSON's more readable cousin, written for humans to edit by hand — indentation replaces {} and [], and quotes on strings are usually optional. safe_load and safe_dump are the same round trip json.loads/dumps do, just for this format, and "safe" specifically means it refuses to construct arbitrary Python objects from tags in the input — unlike the older yaml.load without a restricted Loader.

yaml

service: billing-api
replicas: 3
env:
  LOG_LEVEL: info
  TIMEOUT_MS: 3000
regions:
  - us-east-1
  - eu-west-1

What we're doing: Serialize a config dict to YAML with safe_dump, parse it back with safe_load, and confirm safe_load rejects an unsafe type tag.

yaml_roundtrip.pypython

import yaml

config = {
    "service": "billing-api",
    "replicas": 3,
    "env": {"LOG_LEVEL": "info", "TIMEOUT_MS": 3000},
    "regions": ["us-east-1", "eu-west-1"],
}

text = yaml.safe_dump(config, sort_keys=False)
print(text)

loaded = yaml.safe_load(text)
print(loaded == config)

yaml.safe_load("bad: !!python/object/apply:os.system ['echo hi']")

9: sort_keys=False preserves insertion order in the output instead of alphabetizing keys.
12: safe_load parses the YAML text back into the same nested dict — the round trip is lossless for these types.
15: The !!python/object/apply tag asks the loader to call a Python function while parsing — safe_load refuses and raises instead of running it.

Output

service: billing-api
replicas: 3
env:
  LOG_LEVEL: info
  TIMEOUT_MS: 3000
regions:
- us-east-1
- eu-west-1

True
Traceback (most recent call last):
  ...
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:os.system'

Why this works: safe_dump only ever emits plain YAML scalars, mappings, and sequences, so its output round-trips exactly through safe_load. The !!python/object/apply tag is PyYAML's syntax for "construct this Python object" — safe_load's restricted Loader recognizes that tag as unsafe and raises ConstructorError instead of executing it, which is the entire reason to prefer it over plain yaml.load.

Remember: yaml.safe_load/safe_dump for the JSON-like round trip; never plain yaml.load on untrusted input — it can construct arbitrary Python objects via type tags.

Binary formats, cost, and evolution

The binary formats outside the stdlib and when each fits, the real cost of encoding and decoding, and how to change a schema without breaking readers still on the old one.

Binary serialization formats: MessagePack, Protocol Buffers, Avro

standardintermediate

MessagePack, Protocol Buffers, and Avro all encode data as compact binary instead of text like JSON. MessagePack is a drop-in binary JSON with no schema; Protocol Buffers and Avro both use a schema, which is what makes safe format evolution possible.

Think of it as

JSON is a form anyone can read by eye; these three are the same information vacuum-packed into bytes, trading human-readability for smaller size and faster parsing. MessagePack keeps JSON's "no schema, just send the data" style. Protocol Buffers and Avro instead agree on a schema up front — like a shared form template both sides already have — so the message itself does not need to repeat every field name.

text

MessagePack   — no schema,        binary JSON             — msgpack.org
Protocol Buffers — schema (.proto), numbered fields, typed  — protobuf.dev
Avro          — schema (JSON),    fields resolved by name  — avro.apache.org

MessagePack vs. Protocol Buffers vs. Avro

MessagePack vs. Protocol Buffers vs. Avro
Format	Schema	Typical use	Source
MessagePack	None — same dynamic model as JSON	Fast, compact drop-in replacement for JSON (caches, queues)	msgpack.org
Protocol Buffers	Required (.proto, compiled)	Typed cross-language RPC — gRPC service contracts	protobuf.dev
Avro	Required (JSON-defined schema)	Schema evolution in data pipelines — Kafka, Hadoop	avro.apache.org

Together

python

# Conceptual — no wire-format code executed for this comparison-tier concept.
# MessagePack: same shape as json, just binary and schema-free
# msgpack.packb({"user_id": 4821, "active": True})   -> bytes
# msgpack.unpackb(data)                              -> {"user_id": 4821, "active": True}

# Protocol Buffers: message defined in a .proto file, compiled to Python classes
# user = user_pb2.User(user_id=4821, active=True)
# data = user.SerializeToString()

# Avro: schema is itself JSON, shipped alongside or with the data
# schema = {"type": "record", "name": "User",
#           "fields": [{"name": "user_id", "type": "int"}, {"name": "active", "type": "boolean"}]}

Remember: MessagePack = schema-free binary JSON; Protocol Buffers = typed schema with numbered fields for RPC; Avro = schema-based format built around resolving reader/writer differences by name.

Serialization/deserialization costs

standardintermediate

Every format spends CPU time encoding and decoding, and bytes on the wire or on disk — text formats like JSON and YAML are readable but larger and slower to parse; binary formats like pickle, MessagePack, Protobuf, and Avro are smaller and faster but not human-readable.

Think of it as

Think of it as packing a box. JSON/YAML write the packing list out in full English on the outside — easy to read, takes more space. A binary format like MessagePack or Protobuf writes it in a dense code both sides already agreed on — smaller box, faster to pack and unpack, but unreadable without the codebook (the schema, or the format spec).

python

import json, pickle

payload = {"user_id": 4821, "name": "Priya Shah", "active": True, "roles": ["admin", "editor"]}
len(json.dumps(payload).encode("utf-8"))   # text format: readable, larger
len(pickle.dumps(payload))                 # binary format: not readable, comparable or smaller

What we're doing: Measure the actual encoded byte size of the same payload under json (text) vs. pickle (binary) on this interpreter.

size_comparison.pypython

import json, pickle

payload = {"user_id": 4821, "name": "Priya Shah", "active": True, "roles": ["admin", "editor"]}

json_bytes = json.dumps(payload).encode("utf-8")
pickle_bytes = pickle.dumps(payload)

print("json bytes:", len(json_bytes))
print("pickle bytes (protocol", pickle.DEFAULT_PROTOCOL, "):", len(pickle_bytes))

5: json.dumps produces readable text — every key name is spelled out in full on every encode.
6: pickle.dumps uses opcodes and a memo table instead of repeating field names in the same way JSON text does.

Output

json bytes: 85
pickle bytes (protocol 5 ): 88

Why this works: For this small, simple payload, json and pickle land at nearly the same size — the size advantage of a binary format widens with larger payloads, repeated field names across many records, or a schema-based format (Protobuf/Avro) that omits field names from the wire format entirely, none of which this single small dict demonstrates. The honest takeaway from this run is: measure your actual payload, do not assume a format is smaller without checking.

Remember: Serialization cost is real CPU + bytes spent per call — worth measuring on a hot path with your actual payload, not assumed from a general "format X is faster" claim.

See also: json · pickle · binary serialization formats

Schema evolution and backward compatibility

standardintermediate

Schema evolution is changing what a serialized record looks like — adding, removing, or renaming a field — without breaking readers still using the old schema, or old data still on disk written with it. Backward compatibility means a new reader can still read old data.

Think of it as

Think of a schema as a form both sides agree on. Schema evolution is redesigning that form later — adding a field, say — while some people still hold the old paper version. A change is compatible if the new reader can still make sense of an old form (backward compatible) and, ideally, an old reader does not choke on a new one (forward compatible) — usually by treating a field it does not recognize as optional and ignoring it, rather than failing.

text

Safe:     add optional field with a default
Safe:     remove a field nothing still reads
Breaking: remove a field an existing reader expects
Breaking: rename a field
Breaking: change a field's type

Safe vs. breaking schema changes

Safe vs. breaking schema changes
Change	Safe?	Why
Add a new optional field with a default	Safe	old data simply gets the default value when read by the new schema
Remove a field nothing else still reads	Usually safe	readers that never used it are unaffected
Remove a field an existing reader expects	Breaking	that reader either errors or silently loses data it needed
Rename a field	Breaking (looks like remove + add)	a name-based resolver (Avro) sees the old name gone and a new one appear
Change a field's type (e.g. str -> int)	Breaking	old data in the old type does not parse cleanly as the new type
Reuse a retired Protobuf field-tag number	Breaking	old data tagged with that number is now misread as the new field

Together

python

# Conceptual — the pattern any reader should follow when a schema evolves.
# Old data: {"user_id": 4821, "name": "Priya Shah"}
# New schema adds an optional field with a default, not a required one:

def read_user(record: dict) -> dict:
    return {
        "user_id": record["user_id"],
        "name": record["name"],
        "email_verified": record.get("email_verified", False),  # default for old data
    }

Remember: Add optional fields with defaults; never remove or rename a field a live reader still expects — that is what breaks backward compatibility, not adding to a schema.

See also: binary serialization formats · json