torch.multiprocessing Pool results fail with 'unable to allocate shared memory'
When a torch.multiprocessing Pool returns torch tensors, they are moved into shared memory (/dev/shm files) to cross the process boundary. In containers where /dev/shm is small (64 MB is the Docker default), map() fails with MaybeEncodingError: 'Error sending result' and the reason RuntimeError('unable to allocate shared memory(shm) ... No space left on device'). The traceback prints the whole result list first, so grep for 'Reason:' to find the cause. For a repro harness you can convert tensors to numpy inside the worker before returning (values unchanged). Otherwise enlarge /dev/shm (--shm-size) or try torch.multiprocessing.set_sharing_strategy('file_system').
Reproduced with torch 2.15.0+cpu and CPython 3.12.16 on a 64 MB /dev/shm. A fork Pool(2) mapping 8 tasks that each return a 20 MB tensor fails with MaybeEncodingError, '... Reason: RuntimeError('unable to allocate shared memory(shm) ... No space left on device (28)')'. Returning .numpy() instead works. Two corrections. (1) set_sharing_strategy('file_system') did not help: the same map failed with the same error, because that strategy also places its files in /dev/shm. Only the cleanup mechanism differs. So enlarging /dev/shm or converting to numpy remains the fix. (2) This is not specific to torch.multiprocessing: a plain multiprocessing Pool failed the same way, because importing torch registers its tensor reducers with multiprocessing's pickler. The limit concerns how many results are alive at once: 2 results of 20 MB went through, 8 did not.