beginnerintermediateadvanced

Python · Concurrency and Parallelism

Multiprocessing

Concepts: 2

Real, separate OS processes — each with its own interpreter and memory, so the GIL does not apply across them — for genuine CPU parallelism, and the pickling cost every value pays to communicate between them.

Python overview

Learn Python - Full Course for Beginners [Tutorial]freeCodeCamp.org

Processes and their boundary

Running real, separate processes for CPU-bound work, and what it costs to move data across the boundary between them.

multiprocessing and Process

coreintermediate

multiprocessing runs code in separate OS processes instead of threads. Each process gets its own Python interpreter and its own memory — so unlike threading, multiple processes really do run CPU-bound work at the same time.

Think of it as

Threading is several cooks sharing one kitchen, taking turns at the one stove (the GIL). Multiprocessing is giving each cook their own separate kitchen — they can all cook at once, but now ingredients (data) have to be physically carried between kitchens instead of just handed across the counter.

python

import multiprocessing as mp

def worker(n):
    return n * n

if __name__ == "__main__":
    p = mp.Process(target=worker, args=(4,))
    p.start()
    p.join()

What we're doing: Run the same CPU-bound function serially (twice, back to back) and then across a 2-process Pool, timing both, to see multiprocessing actually use more than one CPU core.

multiprocessing_speedup.pypython

import multiprocessing as mp
import time

def cpu_heavy(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

if __name__ == "__main__":
    N = 5_000_000

    t0 = time.perf_counter()
    r1 = cpu_heavy(N)
    r2 = cpu_heavy(N)
    t_serial = time.perf_counter() - t0

    t0 = time.perf_counter()
    with mp.Pool(processes=2) as pool:
        results = pool.map(cpu_heavy, [N, N])
    t_parallel = time.perf_counter() - t0

    print(f"serial: {t_serial:.3f}s, parallel(2 procs): {t_parallel:.3f}s, results match: {results == [r1, r2]}")

16: pool.map hands one N to each of the 2 worker processes and runs both cpu_heavy calls genuinely at the same time, on separate cores.
17: Starting worker processes and pickling the function/arguments to them is real, measurable overhead — it eats into the speedup a short-lived task would otherwise show.

Output

serial: 0.554s, parallel(2 procs): 0.514s, results match: True

Why this works: The two runs finish in close to the same wall-clock time here, not because multiprocessing failed, but because starting two fresh processes has real fixed overhead (each gets its own interpreter startup) that eats most of this task's modest speedup. For CPU work that runs long enough to dwarf that startup cost, the parallel version pulls ahead — this is the honest, measured shape of that tradeoff, not an idealized 2x.

A target function that isn't defined at module level can't be sent to a child process

Wrong

python

import multiprocessing as mp

def outer():
    def local_worker(q):
        q.put("should never get here")

    q = mp.Queue()
    p = mp.Process(target=local_worker, args=(q,))
    p.start()   # fails here
    p.join()

if __name__ == "__main__":
    outer()

Better

python

import multiprocessing as mp

def local_worker(q):   # module-level, importable by name
    q.put("hello from child process")

def outer():
    q = mp.Queue()
    p = mp.Process(target=local_worker, args=(q,))
    p.start()
    p.join()
    print(q.get())

if __name__ == "__main__":
    outer()

What you see: _pickle.PicklingError: Can't pickle local object <function outer.<locals>.local_worker at 0x...>

Why: A Process target is sent to the child by pickling it — but pickle stores a function by its module path and name, not its actual code. A function defined inside another function has no importable module-level name, so pickle cannot reconstruct a reference to it in the child process. Moving the target to module level fixes it immediately.

Each process gets its own interpreter and memory

Process A

own interpreter, own GIL, own memory

pickle over a pipe

the only way data crosses

Process B

own interpreter, own GIL, own memory

Pool methods for handing out work

Pool methods for handing out work
Method	Argument shape	Returns
pool.map(fn, iterable)	one argument per call	list of results, in order
pool.starmap(fn, iterable_of_tuples)	multiple arguments per call, unpacked	list of results, in order
pool.apply_async(fn, args)	one call, non-blocking	an AsyncResult — call .get() to block for it

Together

python

import multiprocessing as mp

def add(a, b):
    return a + b

if __name__ == "__main__":
    with mp.Pool(processes=2) as pool:
        print(pool.starmap(add, [(1, 2), (3, 4)]))

Remember: multiprocessing runs real, separate processes — genuine CPU parallelism, at the cost of pickling data across the process boundary and real startup overhead.

IPC, serialization, and shared memory

standardintermediate

Separate processes share no memory, so passing data means IPC — Queue and Pipe are the built-in ways. Every value sent must be pickled first, which is why lambdas and local functions cannot cross a process boundary.

Think of it as

Two processes are like two offices in different buildings — nothing on one desk is visible from the other. A Queue is a courier: to send something, you have to write it down in a form the courier can carry (pickle it), and the other office reads that form back into an object (unpickles it). shared_memory is the rare case of both offices getting a key to the same actual filing cabinet, skipping the courier entirely.

python

import multiprocessing as mp

def worker(q):
    q.put("hello from child process")

if __name__ == "__main__":
    q = mp.Queue()
    p = mp.Process(target=worker, args=(q,))
    p.start()
    msg = q.get()
    p.join()

What we're doing: Send a real value to a child process over a Queue, confirm a lambda genuinely cannot be pickled, and use shared_memory to write a byte from one handle and read it back from a second handle to the same buffer.

ipc_and_shared_memory.pypython

import multiprocessing as mp
import pickle
from multiprocessing import shared_memory

def worker_queue(q):
    q.put("hello from child process")

if __name__ == "__main__":
    q = mp.Queue()
    p = mp.Process(target=worker_queue, args=(q,))
    p.start()
    msg = q.get()
    p.join()
    print(f"Queue IPC received: {msg!r}")

    try:
        pickle.dumps(lambda x: x)
    except (pickle.PicklingError, AttributeError, TypeError) as e:
        print(f"lambda pickle error: {type(e).__name__}: {e}")

    shm = shared_memory.SharedMemory(create=True, size=10)
    shm.buf[0] = 42
    shm2 = shared_memory.SharedMemory(name=shm.name)
    print(f"shared memory: wrote 42, read back via second handle: {shm2.buf[0]}")
    shm2.close()
    shm.close()
    shm.unlink()

17: A lambda has no module-level name pickle can store — this fails before the value ever tries to leave the process.
24: shm2 opens the SAME underlying buffer shm created, by name — writing through shm.buf and reading through shm2.buf touches identical memory, no pickling involved.

Output

Queue IPC received: 'hello from child process'
lambda pickle error: PicklingError: Can't pickle <function <lambda> at 0x0000022C79C72B90>: it's not found as __main__.<lambda>
shared memory: wrote 42, read back via second handle: 42

Why this works: The Queue value round-trips fine because a plain string is trivially picklable. The lambda fails for exactly the reason the mental model predicts — pickle looks for it at __main__.<lambda> and finds nothing, since a lambda has no real name. The shared_memory write is visible through a completely separate handle (shm2) because both point at the same OS-level buffer — this is the one multiprocessing mechanism that is not, underneath, a pickle-and-copy.

Forgetting to unlink shared memory leaks it past process exit

Wrong

python

from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=10)
shm.buf[0] = 42
# ... use it ...
shm.close()   # detaches this handle, but the block itself is never freed

Better

python

from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=10)
try:
    shm.buf[0] = 42
    # ... use it ...
finally:
    shm.close()     # every process that opened it must close its own handle
    shm.unlink()     # exactly ONE process calls this, to actually free the block

What you see: The OS-level shared memory segment keeps existing after every Python process using it has exited — visible as a leaked resource, sometimes surfacing as a "file exists" error on the next run trying to create a segment with the same name.

Why: close() only detaches the current process's handle to the shared block; it does not destroy the block itself. unlink() is what actually releases the underlying OS resource, and it should be called exactly once — by whichever process is logically responsible for the block's lifetime, typically the one that created it.

A Queue pickles every item it carries

Process A

q.put(value)

pickle.dumps → bytes → pickle.loads

the actual transport

Process B

q.get() returns the copy

What can and cannot cross a process boundary

What can and cannot cross a process boundary
Value	Picklable?	Why
A module-level function	yes	pickle stores it by module path + name, re-imports it in the child
A lambda	no	has no module-level name pickle can reference
A function defined inside another function	no	same problem — no importable path
A plain dict, list, str, int, dataclass	yes	pickle knows how to serialize built-in and simple types
An open file handle or socket	no	an OS resource with no meaningful serialized form

Remember: Everything sent between processes is pickled by default — lambdas, local functions, open handles cannot cross. shared_memory is the deliberate exception.