Python Multiprocessing, Revisited: Fork vs Spawn

A practical guide to choosing — and safely tuning — Python’s start methods across Linux, macOS, and Windows.

Understand Python multiprocessing start methods (fork, spawn, forkserver), copy-on-write, pickling costs, and real-world tuning tips for faster, safer parallel code.

You’ve probably “fixed” the GIL with multiprocessing — then hit weird hangs, memory spikes, or sluggish startups. Let’s be real: the start method you choose can make or break performance. Here’s how to pick wisely and avoid landmines.

The TL;DR mental model

• fork is fast to start (copy-on-write) but fragile with threads and some native libs.
• spawn is safer and cross-platform, but slower (fresh interpreter + pickle).
• forkserver splits the difference: safer than fork; faster and lighter than pure spawn when processes are short-lived and numerous.

If you write data science, web, or ML code that forks a process after threads or GPU contexts exist, expect surprises. Choose deliberately.

Start methods in one picture

Start Method Decision Flow
┌─────────────────────────────────────────────────────────┐
│ Need cross-platform safety (Windows/macOS)?  ──► SPAWN  │
│ Using threads / async runtimes / GPU handles? ──► SPAWN │
│ Many short tasks, want speed, Linux only?      ──► FORK │
│ Want safer preloading of modules on Linux?     ─► FORKSERVER
└─────────────────────────────────────────────────────────┘

Why fork feels fast — and when it bites

Copy-on-write (CoW)

On POSIX, fork clones the parent’s address space lazily. Big models, configs, or datasets appear “free” to child processes until you mutate memory.

Benefit: Near-zero startup overhead, great for CPU-bound map operations where the parent preloads large read-only data (e.g., embeddings, lookup tables).

Gotchas:

• Threads: If the parent has active threads (BLAS, logging, web servers), the child inherits only one running thread but locks may be held → deadlocks.
• Non-fork-safe libs: OpenMP/BLAS pools, CUDA contexts, random device handles, or network sockets can misbehave.
• Signal handlers & atexit: Inherited state can fire unexpectedly.

Rule: If you fork, fork early (before threads) and keep child initialization simple.

Why spawn feels slow — and why you may still want it

spawn starts a brand-new Python interpreter, imports your target, then pickles the callables/arguments. It’s deterministic and safe with threads/GPU contexts.

Costs:

• Import time on every worker.
• Pickling overhead for large objects.
• More memory (no CoW sharing).

Wins:

• Works the same on Windows (default) and macOS (default since 3.8).
• Avoids most fork-after-threads landmines.
• Predictable in serverless/containers when parents are busy.

Optimization: amortize the startup by long-lived pools and initializer hooks to load heavy models once per worker.

The middle path: forkserver

A small server process is started with fork(). New workers ask the server to fork cleanly (no busy threads yet), then the worker exec()s Python. Safer than raw fork, often faster than pure spawn for lots of tiny tasks.

Get Nexumo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Best when: Linux; you want “mostly safe” creation speed and consistent behavior.

Practical setup (do this first)

from multiprocessing import set_start_method, get_start_method

if __name__ == "__main__":
    # Choose explicitly; fail fast if already set.
    try:
        set_start_method("spawn")      # or "fork", or "forkserver"
    except RuntimeError:
        pass
    print("start method:", get_start_method())

• Linux CLI/data pipelines: prefer "fork" or "forkserver".
• macOS & Windows apps, web, notebooks: "spawn" is the safe default.

Real-world patterns

1) Heavy model, many short tasks (Linux)

Preload once; use fork for CoW; avoid mutation.

# parent.py
import numpy as np
BIG = np.memmap("embedding.bin", dtype="float32", mode="r")

def score(idx):
    v = BIG[idx]          # read-only slice: stays CoW-shared
    return float(v.sum())

# Speed: set "fork" before importing this module's pool.

Beware: Any in-child writes to BIG break sharing and balloon memory.

2) Web server + workers (threads active)

Use spawn to avoid deadlocks with threaded servers or BLAS.

from concurrent.futures import ProcessPoolExecutor

EXEC = ProcessPoolExecutor(
    max_workers=4,
    mp_context=None  # honors global start method (spawn recommended)
)

def cpu_heavy(x): ...

Tip: Keep the pool global and reuse it to amortize spawn cost.

3) Safer speed on Linux with many tiny tasks

Use forkserver and preload modules to trim per-child import time.

from multiprocessing import set_start_method, get_context

if __name__ == "__main__":
    set_start_method("forkserver", force=True)
    ctx = get_context("forkserver")
    with ctx.Pool(processes=8, maxtasksperchild=1000) as pool:
        print(list(pool.imap_unordered(abs, range(-10, 10))))

Data passing: pickle pain vs. zero-copy options

The default: Pickle

Simple, but expensive for big NumPy arrays or Pandas frames.

Faster: multiprocessing.shared_memory (Py3.8+)

Share large arrays without copying.

# parent.py
import numpy as np
from multiprocessing import shared_memory, Process

arr = np.random.rand(1_000_000).astype("float64")
shm = shared_memory.SharedMemory(create=True, size=arr.nbytes)
back = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
back[:] = arr  # copy once into shared segment

def worker(name, shape, dtype):
    import numpy as np, multiprocessing.shared_memory as sm
    shm = sm.SharedMemory(name=name)
    a = np.ndarray(shape, dtype=dtype, buffer=shm.buf)
    print(a[:3].mean())
    shm.close()

if __name__ == "__main__":
    p = Process(target=worker, args=(shm.name, arr.shape, arr.dtype))
    p.start(); p.join()
    shm.close(); shm.unlink()

Rule of thumb: For multi-MB arrays reused across tasks, shared memory pays off.

Pools, executors, and task hygiene

• maxtasksperchild: recycle workers to mitigate memory leaks in native libs.
• initializer: load models/dicts once per worker; store in module globals.
• Batching: Prefer imap_unordered(chunksize=k) to reduce scheduler chatter.
• Backpressure: For streaming, pair a bounded multiprocessing.Queue with producer checks to avoid OOM.
• Graceful stop: Use sentinels (e.g., None) or pool.close(); pool.join().

Measuring what matters (quick bench)

import time, os
from multiprocessing import get_context

def work(i): s=0
for _ in range(200_000): s+=i*i
return s

def bench(method):
    ctx = get_context(method)
    t0 = time.perf_counter()
    with ctx.Pool(4) as p:
        out = list(p.imap_unordered(work, range(16), chunksize=2))
    dt = (time.perf_counter() - t0)*1000
    print(f"{method:10} {dt:7.1f} ms")

if __name__ == "__main__":
    for m in ("spawn","forkserver","fork"):
        try: bench(m)
        except Exception as e: print(m, "->", e)

Run on your machine; you’ll see spawn’s startup tax and when fork/forkserver shine.

Common failure modes (and fixes)

• Hangs after fork: You forked with active threads. Use spawn or forkserver.
• Random crashes in NumPy/BLAS: Thread pools + fork. Set OPENBLAS_NUM_THREADS=1 or choose spawn.
• Huge memory per worker: You mutated data that was CoW-shared. Make read-only or use shared memory.
• Slow first task on spawn: Heavy imports; use pool initializer to preload once.
• Pickling errors (“can’t pickle local object”): Move callables to top level; avoid lambdas/closures.

Reality check: the GIL and when not to use processes

If your workload is I/O-bound, asyncio + threads may beat processes with less overhead and simpler data sharing. Save multiprocessing for CPU-bound or isolation-critical tasks.

A safe default recipe (works almost everywhere)

# safe_pool.py
from multiprocessing import set_start_method, get_context
from concurrent.futures import ProcessPoolExecutor

def make_pool(workers=4):
    try: set_start_method("spawn")
    except RuntimeError: pass
    ctx = get_context("spawn")
    return ProcessPoolExecutor(max_workers=workers, mp_context=ctx)

Use this in web apps, CLIs, and notebooks when you value predictability over raw startup speed.

Conclusion

Multiprocessing isn’t just “create more processes.” It’s choosing the right start method for your environment and libraries, then moving data the smart way. Fork is a rocket — use it on the right launch pad. Spawn is a truck — slower off the line but safe anywhere. Forkserver is the shuttle bus between the two.

If this helped, drop a comment with your stack (OS, NumPy/BLAS, web framework), and I’ll suggest a tailored start-method plan. Follow for more pragmatic Python performance guides.