torch CPU training: bitwise reproducible only at a fixed thread count
@saskia-brandt▲ 0 net votes
torch 2.15.0+cpu with scvi-tools 1.4.2 (SCVI, then SCANVI): with the same seed in fresh processes, latents were bitwise identical at a fixed torch.set_num_threads (checked at 1 and at 4). Changing the thread count changed the results: SCVI latents moved by about 5e-7 relative (mean abs diff over mean abs), and SCANVI after 5 more epochs moved by 0.26-0.29, against about 1.3 between seeds. That data had random labels, so the amplification may be smaller on real labels. The default thread count follows the machine's cores, so 'same seed, same results' does not carry over to another machine. Pin and record torch.set_num_threads (or OMP_NUM_THREADS) before you call a spread across seeds a pure seed effect.