seed + worker_id generators replay every epoch with non-persistent DataLoader workers
A common pattern is to lazily build a per-worker generator as Generator(seed + get_worker_info().id) inside a collator or Dataset. It gives distinct streams across workers, but with persistent_workers=False (the default) the workers are recreated each epoch and rebuild the same generators. Measured with torch 2.15, 4 workers, 3 epochs, no shuffle: every item got the identical random mask in every epoch (64/64), under fork, spawn and forkserver alike. With shuffle the per-item draws differ, but each epoch reuses exactly the same multiset of draws. num_workers=0 or persistent_workers=True do not replay. To vary draws per epoch, mix the epoch or torch.initial_seed() (which DataLoader changes per epoch) into the seed instead of a constant.
Reproduced on torch 2.15.0+cpu, NumPy 2.5.4, CPython 3.12.16: 32 items, batch_size=1, 4 workers, 3 epochs, Dataset with a lazy Generator(PCG64(1234 + worker_id)). Without shuffle, every item got the identical draw in all 3 epochs (32/32; 32 distinct values out of 96), under fork, spawn and forkserver. With shuffle=True, 0/32 items matched across epochs, but epoch 0 and epoch 1 had exactly the same sorted multiset of draws. persistent_workers=True gave 96/96 distinct. The suggested fix also checks out: seeding the lazy generator from torch.initial_seed() % 2**32 inside the worker gave 96/96 under fork and spawn. That value already differs per worker and per epoch (base_seed + worker_id), so it is not necessary to add the worker id again.