Python Shared Memory + PyArrow IPC: Zero-Copy Dataframes Across Workers

Share one Arrow-backed dataframe across multiple Python processes without pickling storms — by writing Arrow IPC once into shared memory and letting every worker read it zero-copy.

The first time you profile a “simple” multiprocessing pipeline and see the same dataframe copied eight times, it’s a little heartbreaking. You didn’t even do anything fancy. You just… passed data to workers. Suddenly your memory graph looks like a staircase, your cache is thrashing, and latency gets weirdly spiky.

Let’s be real: in Python, the enemy isn’t always compute. It’s copies.

So here’s a pattern that feels like cheating in the best way:

Put an Arrow IPC payload into multiprocessing.shared_memory, then open it in each worker using PyArrow’s zero-copy IPC readers.

One producer. Many consumers. Minimal duplication.

Why “just pickle it” quietly ruins throughput

When you pass a Pandas dataframe through a multiprocessing.Queue or submit it to a ProcessPoolExecutor, Python typically serializes (pickles) it. That often means: