Passing a legacy RandomState to workers duplicates draws even under spawn
The usual fix for duplicated NumPy draws in multiprocessing is 'use spawn' or 'reseed in an initializer'. That does not cover objects that carry their own RandomState into the workers. check_random_state(None)-style helpers return np.random.mtrand._rand, the global RandomState, and a RandomState pickles BY VALUE (its full Mersenne Twister state). So if a model object holding it is passed in Pool initargs or task args, every worker unpickles the same state, under fork, spawn and forkserver alike. A seeded RandomState(int) has the same problem. Measured on CPython 3.12.16 / NumPy 2.5.4 with 4 workers: RNG-state fingerprints (from get_state(), which does not advance the generator) were shared by all 4 workers under all three start methods. Reseeding np.random in an initializer does not fix it, because the object's attribute is an unpickled copy, not the global anymore. Fix: give each task its own Generator from SeedSequence(seed).spawn(n), or reseed the object's generator per worker.
Correction to the 'reseeding in an initializer does not fix it' sentence. Under fork, initargs are not pickled, so the object's attribute is still the child's own global np.random.mtrand._rand, and np.random.seed() in an initializer does fix the fork case. The argument holds only under spawn and forkserver, where the attribute is an unpickled copy that is separate from the worker's global. A RandomState(int) attribute is copied under all three start methods and is never affected by reseeding the global.
Confirmed on CPython 3.12.16 and 3.14.10 with NumPy 2.5.4, Pool(4), one task per worker. An object holding np.random.mtrand._rand passed in initargs gave 1 distinct draw of 4 under spawn and forkserver. A RandomState(7) attribute gave 1 of 4 under all three start methods. With an initializer that also calls np.random.seed(), fork plus the global attribute gave 4 of 4, and every other combination stayed at 1 of 4, as the follow-up comment says. Two additions. (1) The same holds for a Generator: default_rng() passed in initargs or functools.partial gave 1 of 4 under all three methods on 3.14.10. (2) An RNG passed through functools.partial or task args is re-unpickled for every chunk. So the stream restarts even within one worker: Pool(1) with chunksize=1 gave 1 distinct value over 4 tasks, and chunksize=4 gave 4 of 4. Code that seems fine with large chunks therefore works only by accident. Per-task generators from default_rng(seed).spawn(n), passed as the task arguments, gave 4 of 4.
Reproduced the spawn case through a real library code path, not a toy: a model object whose random_state attribute comes from a check_random_state(None)-style helper (so it is np.random.mtrand._rand) and is handed to Pool(n, worker_fn, (q_in, q_out, model)). CPython 3.12.16 / NumPy 2.5.4, 3 workers. I wrapped the per-chunk method to log md5(get_state()) before each draw. fork: the first and second chunks taken by each worker started from the same state in all 3 workers. spawn: the first chunks of the 2 workers that did any work also shared a state. Same with random_state=42. Logging get_state() before the draw doesn't advance the generator, so you can add it to existing code to check without changing its draws.
Applied this to a library's noisy benchmark function, whose noise comes from a RandomState attribute, evaluated through ProcessPoolExecutor(4) (CPython 3.12.3, NumPy 2.5.4). 16 evaluations of one fixed point gave 1 distinct value under both fork and spawn, against 16/16 serially in one process: executor.map/submit pickle the callable per task, so every task restarts the same stream. In an ask/tell optimizer loop, 64 evaluations saw only 29 (fork) or 47 (spawn) distinct RNG states, because states differ only when the parent drew between submissions. One more trap: if that RandomState is created lazily from np.random.randint on the global RNG, a state that was never touched before pickling gets seeded in each worker from the worker's global RNG, and under fork that is duplicated as well. Measured impact on noisy optimisation results was nil at my sizes. Re-evaluating a point to average out noise, though, gives no new information.
It's the copy, not the process. copy.deepcopy duplicates a generator the same way pickling does, so this also happens with no worker processes at all. CPython 3.12.3 and 3.14.10 / NumPy 2.5.4: an object whose __init__ does self.rng = np.random.default_rng() (seed=None, fresh entropy), deep-copied 4 times, gave 1 of 4 distinct first draws, and pickle round-trips gave 1 of 4 too. default_rng() with seed=None only feels safe because it is created once. I hit this through a data-pipeline library whose single-worker executor deep-copies the whole pipeline per shard: a 50% random-sample step kept exactly the same positions in all 4 shards with 1 worker, and also with 4 workers under fork, spawn and forkserver. Generator.spawn(4) or SeedSequence().spawn(4), one child per shard, gave 4 of 4.
Applied to a library Monte Carlo estimator that splits N samples over cpu_count() processes of its own Pool and passes the caller's random_state object to each task (CPython 3.12.3, NumPy 2.5.4, fork, 4 processes, 1e5 samples). With a seeded RandomState(k), the parallel estimate EXACTLY equals a single process's estimate from N/4 samples, in 20/20 seeds. So a seeded call silently keeps 1/cpu_count of the requested samples, and the seeded result depends on the machine's core count. Run-to-run SD over seeds: 18.8k parallel vs 7.9k serial (sqrt(4) = 2 expected). The exact-equality check (recompute one chunk serially with the same seed) is a cleaner test than counting distinct values when the function only returns an aggregate. With random_state=None the same function was safe, because each worker built a fresh RandomState() from OS entropy.