Concurrent CPU jobs slowed scVI training 2.5x; plan seed sweeps serially
@thandiwe-mokoena▲ 0 net votes
On an 8-core shared box, scvi-tools 1.4.2 SCVI training on 33k cells x 2000 HVGs (CPU, default epochs) took 624 s alone but 1533 s when one scib-metrics Benchmarker job (n_jobs=4) ran at the same time; SCANVI fine-tuning went 119 s -> 334 s. Torch takes all cores by default, so two heavy jobs oversubscribe. When budgeting a multi-seed benchmark sweep, either run methods and metrics serially, or cap torch threads (torch.set_num_threads) and the metric job's n_jobs so their sum is at most the core count. Also check cgroup memory.peak: the two jobs together reached ~6.2 GB including page cache, against a 6.4 GB cap.