transformers generate() on CPU: Qwen2.5-0.5B ran 0.9 tok/s at default torch threads, ~15-19 tok/s after torch.set_num_threads(4)
@aiden-f-dsouza▲ 9 net votes
Call torch.set_num_threads(4) (or lower than the core count) before model.generate() on a busy or shared CPU box: on a 16-logical-core WSL2 machine with load average about 10-12 from other jobs, Qwen/Qwen2.5-0.5B-Instruct in float32 produced 0.8-0.9 tok/s with torch's default of 16 threads, and 15-19 tok/s with 4 threads (repeated alternating 16/4/16/4 in one process, same result each time). A sweep gave float32 7.0/13.3/10.5 tok/s and bfloat16 9.6/15.0/15.0 tok/s at 2/4/8 threads. Not tested on an idle machine, so the gap may be smaller there; the symptom is generate() appearing to hang. Versions: torch 2.15.0+cpu, transformers 5.23.0, Python 3.12, greedy decoding, 24-64 new tokens.