Architecture · What a 5-Year Python Engineer Should Be Able to Explain · Python

What makes a Python service horizontally scalable

standardadvanced

A Python service becomes horizontally scalable by removing everything that pins a request to one specific process: no in-memory session state (use an external store), no module-level mutable globals holding request-specific data, and no assumption that "the next request" lands on the same worker as the last one — every worker process (across every machine) must be able to serve any request identically.

Think of it as

A stateful worker is a specific employee who remembers your conversation — if they go home, the next employee has no idea who you are. A horizontally scalable worker is any interchangeable employee at any counter, because everything they need to help you (your session, your cart) is written down in a shared filing system (Redis, the database) any counter can read. Adding more counters (worker processes, more machines) only helps once no counter is special.

python

# NOT scalable: state trapped in one worker process's memory
_cart_cache = {}   # module-level dict -- only THIS worker process can see it

# Scalable: state lives in an external store every worker can reach
def get_cart(user_id, redis_client):
    return redis_client.get(f"cart:{user_id}")

What we're doing: Contrast a stateful handler that only works if the same worker serves every request from a user, against a stateless handler backed by an external store that any worker can serve identically.

horizontal_scalability.pypython

_local_cart_cache = {}   # lives in ONE worker process's memory only

def add_to_cart_not_scalable(user_id, item):
    _local_cart_cache.setdefault(user_id, []).append(item)
    return _local_cart_cache[user_id]


class FakeExternalStore:
    def __init__(self):
        self._data = {}
    def get(self, key):
        return self._data.get(key, [])
    def append(self, key, item):
        self._data.setdefault(key, []).append(item)
        return self._data[key]


shared_store = FakeExternalStore()   # models Redis: reachable from ANY worker

def add_to_cart_scalable(user_id, item, store):
    return store.append(f"cart:{user_id}", item)


print("worker A view:", add_to_cart_scalable("user-1", "widget", shared_store))
print("worker B view (SAME store):", add_to_cart_scalable("user-1", "gadget", shared_store))

1: _local_cart_cache lives in one worker process's memory — a second worker process (a second machine, or even a second process on the same machine) has an entirely separate, empty dict.
15: shared_store models an external store like Redis — every worker process reads and writes the SAME underlying data, which is what makes it safe for any worker to handle any request.

Output

worker A view: ['widget']
worker B view (SAME store): ['widget', 'gadget']

Why this works: Both calls go through add_to_cart_scalable with the same shared_store, modeling two different worker processes (or two different machines) both reaching the same external state — worker B's call correctly sees worker A's earlier item because the cart lives in a store both can reach, not in either worker's own memory. _local_cart_cache would have given worker B an empty list instead, silently losing the first item the moment a second worker (or a restart) was involved.

Assuming "it works on my machine with one worker" proves a service is horizontally scalable

Wrong

python

# tested locally with a single Gunicorn worker (--workers 1)
# module-level cache works fine -- ships to production with --workers 4
# users now randomly "lose" cart items depending on which worker handles each request

Better

python

# test with MULTIPLE workers locally (--workers 4) before shipping,
# and verify state survives a request landing on a DIFFERENT worker
# than the one that handled the previous request from the same user

What you see: The bug is invisible in local development (one worker, every request naturally lands on the same process) and only appears in production once real concurrent traffic gets load-balanced across multiple workers or machines — appearing as intermittent, hard-to-reproduce "missing data" reports.

Why: Statelessness is not observable from a single-worker test — it only fails once a second worker (or a second machine) is actually involved, which is exactly the condition horizontal scaling introduces. Testing locally with more than one worker process is what surfaces this class of bug before production traffic does.

Remember: Externalize every piece of state a worker would otherwise hold in memory (session, cart, job context) into a shared store every worker can reach — statelessness is what makes adding more workers or machines actually increase capacity instead of causing random, worker-dependent bugs.