A process pool looks like a function-call accelerator: submit a callable and arguments, receive a future. The apparent call crosses a boundary that ordinary calls do not. The worker needs a representation of the callable, arguments, exception, and result. It imports modules independently and mutates a different memory space.

Serialization is therefore not executor plumbing. It determines which objects can cross, how expensive each task is, what compatibility means, and whether failures retain useful meaning. A clean worker interface resembles a small local service: explicit input values, explicit output values, stable error categories, and no dependence on the caller's live object graph.

Version note. Experiments were run on CPython 3.14.7 on macOS arm64. CPython uses pickle in process executors, but process creation and transport are platform implementation details. Python 3.14 changed the default POSIX multiprocessing start method from fork to forkserver; macOS and Windows use spawn by default. Query the actual context rather than inferring it from the platform.

Experiment 1: ordinary values cross by copy

from concurrent.futures import ProcessPoolExecutor

def mutate(values):
    values.append('worker')
    return values

def main():
    original = ['parent']
    with ProcessPoolExecutor(max_workers=1) as pool:
        returned = pool.submit(mutate, original).result()
    assert original == ['parent']
    assert returned == ['parent', 'worker']

if __name__ == '__main__':
    main()

The worker reconstructs a list from serialized data. Its mutation is not visible in original; the returned list is serialized back into another parent object. Identity does not cross this boundary.

This separation removes many accidental shared-memory races. It also makes object graphs with hidden resources inappropriate messages. A request carrying a database session, lock, generator, open file, or callback asks serialization to preserve live behavior it cannot meaningfully reproduce.

Use small data classes, tuples, dictionaries of primitive values, bytes, or a documented schema. Version that schema if workers and producers can deploy independently. Even within one release, an explicit message prevents workers from reaching through a giant application object.

Experiment 2: the callable must be importable

from concurrent.futures import ProcessPoolExecutor

def square(value):
    return value * value

def main():
    with ProcessPoolExecutor(max_workers=1) as pool:
        assert pool.submit(square, 7).result() == 49

if __name__ == '__main__':
    main()

Process pools serialize a reference to a normal top-level function by module and qualified name. A lambda or nested function normally cannot be resolved that way. Functions defined interactively also fail because workers need an importable __main__ module.

The guard is not ceremonial. Under spawn-like startup, a worker imports the main module. Without the guard, import can create another pool recursively or rerun side effects. Put application startup behind if __name__ == '__main__' and keep module import declarative.

Python guarantee. ProcessPoolExecutor requires picklable callables and arguments, and the __main__ module must be importable by worker subprocesses. It does not work in the ordinary interactive interpreter.

Experiment 3: pickle answers representation, not trust

import pickle
from dataclasses import dataclass

@dataclass(frozen=True)
class Job:
    tenant_id: int
    samples: tuple[int, ...]

job = Job(17, (2, 4, 8))
payload = pickle.dumps(job, protocol=pickle.HIGHEST_PROTOCOL)
restored = pickle.loads(payload)

assert restored == job
assert restored is not job
print(len(payload))

Pickle preserves a broad range of Python-specific object structures, often by importing classes and invoking reconstruction logic. It is not a safe interchange format. Loading untrusted pickle data can execute arbitrary code. Never accept it from clients, uploads, message buses with untrusted writers, or artifacts whose integrity is uncertain.

Executor transport is intended for cooperating local processes under one application trust boundary. For external boundaries, choose a format with constrained semantics and validate it. JSON, MessagePack, Protocol Buffers, and Arrow solve different schema and performance problems; none should be selected merely because pickle is unsafe.

The printed byte count is environment-specific. It is useful only when comparing representative payloads under a recorded protocol and version.

Experiment 4: exceptions also need a boundary representation

from concurrent.futures import ProcessPoolExecutor

class InvalidRow(Exception):
    pass

def parse(value):
    if value < 0:
        raise InvalidRow(f'negative: {value}')
    return value

def main():
    with ProcessPoolExecutor(max_workers=1) as pool:
        future = pool.submit(parse, -3)
        try:
            future.result()
        except InvalidRow as error:
            assert str(error) == 'negative: -3'
            assert error.__cause__ is not None
        else:
            raise AssertionError('worker should fail')

if __name__ == '__main__':
    main()

The executor tries to preserve the worker exception and sets its cause to an ExecutionFailed summary. If it cannot preserve the original exception, it may preserve the summary instead. Custom exception classes should be importable and constructible from their serialized arguments.

Do not attach giant or unpicklable runtime objects to exceptions. Return structured domain failures when they are expected outcomes, and reserve raised exceptions for task failures. A worker crash, abrupt exit, initializer failure, or deserialization failure can break the pool and surface BrokenProcessPool; callers need a policy distinct from invalid input.

Experiment 5: globals are worker state, not shared state

from concurrent.futures import ProcessPoolExecutor

counter = 0

def increment():
    global counter
    counter += 1
    return counter

def main():
    global counter
    counter = 100
    with ProcessPoolExecutor(max_workers=1) as pool:
        observed = [pool.submit(increment).result() for _ in range(2)]
    assert observed == [1, 2]
    assert counter == 100

if __name__ == '__main__':
    main()

With a spawn-like context, the worker imports the module and starts its own counter. A reused worker retains its own global across tasks, which explains [1, 2]. Do not rely on that persistence for durable state: scheduling may choose another worker, processes can be recycled, and crashes erase memory.

An executor initializer can establish per-worker read-only resources such as a parsed model or client. It should be deterministic and safe to rerun. If initialization fails, pending jobs may fail with BrokenProcessPool.

Fork historically made parent memory appear inherited through copy-on-write, tempting code to depend on warmed globals. That behavior is unavailable under spawn and forkserver and unsafe around many threaded runtimes. Python 3.14's POSIX default change makes import-safe initialization the portable design.

Experiment 6: inspect and choose the context

import multiprocessing as mp

default = mp.get_start_method()
available = mp.get_all_start_methods()

assert default in available
print(default)
print(available)

On the test host this prints spawn. On many Linux CPython 3.14 installations it prints forkserver; Windows supports spawn. The language does not guarantee the same list on every operating system.

Libraries should generally accept a multiprocessing context from callers rather than globally calling set_start_method(). Applications own process policy. ProcessPoolExecutor(mp_context=mp.get_context('spawn')) makes a requirement explicit. Requesting fork in a multithreaded process can produce warnings and unsafe inherited runtime state; Python 3.14 no longer selects it by default anywhere.

Process start policy affects startup latency, resource inheritance, frozen executables, and debugging. Test every explicitly supported platform and deployment packaging mode.

Experiment 7: transfer can dominate computation

import pickle
from time import perf_counter

values = list(range(200_000))

started = perf_counter()
payload = pickle.dumps(values, protocol=pickle.HIGHEST_PROTOCOL)
restored = pickle.loads(payload)
elapsed = perf_counter() - started

assert restored == values
print(len(payload), round(elapsed, 4))

This is not a process-pool benchmark; it isolates one unavoidable category of work. Real submission adds queue synchronization, copying, scheduling, and result transfer. A pool loses when each task serializes a large input to perform a tiny calculation.

Batch many small records into one task, load immutable reference data once per worker, or send compact identifiers that workers resolve locally. Measure end-to-end wall time and peak memory. Pickling often creates temporary byte buffers in addition to live parent and worker objects.

Protocol 5 supports out-of-band buffers, but ProcessPoolExecutor does not turn every numeric object into zero-copy transport automatically. Library-specific shared-memory support can help large arrays; verify ownership and lifetime instead of assuming the word "buffer" removes copying.

Experiment 8: batching changes the scheduling equation

def chunks(values, size):
    for offset in range(0, len(values), size):
        yield values[offset:offset + size]

values = list(range(10))
batches = list(chunks(values, 4))

assert batches == [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9]]
assert [item for batch in batches for item in batch] == values

Batching amortizes submission and serialization overhead but changes responsiveness. Larger batches produce fewer scheduling decisions, less even load balancing, and coarser cancellation. The right size depends on variance as well as average task cost.

Executor.map supports chunksize for process pools; a larger value groups iterable items into fewer submitted tasks. Python 3.14 also adds buffersize, limiting the number of submitted results not yet consumed. chunksize controls task granularity; buffersize controls producer lead. They solve different overload problems.

Tune with representative skew. A final batch containing one pathological item can leave cores idle. Dynamic smaller chunks may beat theoretically lower serialization overhead.

Experiment 9: shared memory needs an ownership protocol

from multiprocessing import shared_memory

block = shared_memory.SharedMemory(create=True, size=4)
try:
    block.buf[:4] = b'PY14'
    attached = shared_memory.SharedMemory(name=block.name)
    try:
        assert bytes(attached.buf[:4]) == b'PY14'
    finally:
        attached.close()
finally:
    block.close()
    block.unlink()

Shared memory avoids serializing the payload bytes, but workers still need metadata such as the block name, shape, data type, and slice. It also reintroduces shared-state problems: concurrent writers need synchronization, readers need a publication protocol, and one owner must unlink the segment.

The resource tracker behavior changed in Python 3.13 with the track parameter, and independently launched processes can have separate trackers. Consult the exact target version before designing lifetime around tracker cleanup. Shared memory is an optimization with a protocol, not a transparent faster list.

A worker interface checklist

Design a process worker like an internal service:

  • Keep the callable top-level, importable, and free of parent-only initialization.
  • Pass immutable explicit data rather than live application objects.
  • Validate messages at the boundary and keep schemas compatible.
  • Make expected domain failures serializable and distinguish worker infrastructure failure.
  • Batch enough work to amortize startup, scheduling, and transfer.
  • Bound submitted work so payloads do not fill parent memory.
  • Initialize expensive read-only state once per worker when measurement justifies it.
  • Choose the process context at the application boundary.
  • Treat shared memory as shared mutable infrastructure with explicit ownership.
  • Make jobs idempotent if a crashed worker may require retry.

Cancellation of a future that is already running does not kill its process operation. Python 3.14 adds terminate_workers() and kill_workers() for forceful executor-wide shutdown, not surgical cancellation of one safe task. Force can leave external effects halfway complete, so durable jobs still need idempotency and reconciliation.

Deployment changes the cost model

Measure pools inside the real container or host limits. A runtime may report the host's CPUs while the workload has a smaller quota, and memory limits may make duplicated interpreter heaps the actual constraint. Python 3.13 changed ProcessPoolExecutor's default worker count to use os.process_cpu_count(), but explicit sizing and deployment measurement remain safer than assuming one worker per visible core.

Long-lived workers can accumulate fragmented heaps, native-library caches, and leaked resources. max_tasks_per_child can recycle workers, but the executor uses a spawn-compatible method when that option is supplied and it is incompatible with fork. Recycling trades memory stability for repeated initialization. Fix leaks first; use recycling when libraries or workload behavior justify it.

Native libraries may start their own thread pools in every process. Four Python workers each starting eight BLAS threads can produce 32 runnable native threads, poor cache behavior, and worse latency. Set native worker limits deliberately and record them with benchmark results.

Compatibility belongs to the message

Pickle is tightly coupled to import paths and Python definitions. Renaming a class or deploying producers and workers from different releases can make stored payloads unreadable. Process-pool messages are transient within one program, but queues, retries, and crash recovery can accidentally make them durable.

If messages survive deployment, choose a stable schema with explicit version fields and migrations. Keep an operation name rather than serializing arbitrary callables. Validate sizes before allocating, reject unknown versions, and preserve an idempotency key across retries. At that point the design may deserve a separate worker service rather than an in-process pool.

Even ephemeral messages benefit from schema tests. Round-trip representative values, verify custom exceptions, and run producer and worker code under the packaging mode used in production. Serialization failures discovered before expensive computation are much easier to recover from than a pool broken halfway through a batch.

Exercises

  1. Submit a nested function and inspect its pickling failure. Move it to an importable module and explain what changed.
  2. Compare one-record and 1,000-record batches for a cheap transform. Report serialization bytes, elapsed time, and peak memory.
  3. Create a custom exception with an unpicklable attribute. Redesign it as a stable domain failure.
  4. Run the context experiment on Linux, macOS, and Windows or CI equivalents. Record Python version and defaults.
  5. Replace a large copied bytes payload with shared memory. Specify creator, reader, synchronization, close, and unlink ownership before coding.

Keep this model

A process executor call is a message exchange disguised as a function call. Values are reconstructed, imports happen in independent interpreters, globals belong to workers, and exceptions cross through a serialization protocol. Startup mode and task granularity materially change both correctness and cost.

Processes are excellent for coarse isolated work when that boundary is embraced. They disappoint when code tries to smuggle a live object graph across or ignores transfer. Design the message first, then measure whether parallel execution pays for sending it.

Primary sources