numba OpenMP layer + fork: multiprocessing.Pool hangs forever, no error
If the parent process has run any numba parallel=True kernel under the 'omp' threading layer (numba picks it when libgomp is present and tbb is not installed), a fork-started worker that runs a parallel numba kernel prints 'Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.' and exits. multiprocessing.Pool silently replaces dead workers and the lost tasks never return, so pool.map/imap block forever at 0% with workers idle in pipe_read. The kernel can come from a dependency you never call directly; a groupby/aggregate library that recently added parallel numba paths did it here. Diagnose with faulthandler.register(signal.SIGUSR1, all_threads=True) plus kill -USR1, then grep stderr for the message and check numba.threading_layer() in the parent. Fixes that worked: NUMBA_THREADING_LAYER=workqueue, or the spawn/forkserver start method. tbb is also fork-safe, but I did not test it.
Reproduced with numba 0.68.1, NumPy 2.5.4, system libgomp, no tbb: one prange kernel run in the parent, then a 2-worker Pool mapping the same kernel. On CPython 3.12.16 with fork: 4x "Terminating: fork() called from a process already using GNU OpenMP, this is unsafe." and map_async(...).get(timeout=30) times out. spawn, forkserver, NUMBA_THREADING_LAYER=workqueue, and not running the kernel in the parent all returned results.
Two additions: 1. ProcessPoolExecutor does not hang: with fork it raises BrokenProcessPool right away. The silent hang is specific to multiprocessing.Pool, which replaces dead workers. 2. On 3.14.10, forkserver is only a fix if the parent's first parallel kernel call happens under if __name__ == "__main__". If the call runs at module level, the 3.14 forkserver preloads __main__, so the server process itself starts GNU OpenMP, and every worker forked from it dies the same way: 4 Terminating lines, Pool hang. The same script with the call moved under the main guard works. spawn and workqueue are the robust fixes.