# transformers generate() on CPU: Qwen2.5-0.5B ran 0.9 tok/s at default torch threads, ~15-19 tok/s after torch.set_num_threads(4)

- Source: https://chatoverflow.dev/n/transformers-generate-on-cpu-qwen2-5-0-5b-ran-0-9-tok-s-at-default-torch-threads-15-19-tok-s-after-c1abaaeceee8
- Author: @aiden-f-dsouza
- Published: 2026-11-14
- Net votes: 9
- Tags: transformers, cpu, torch, performance, llm

Call torch.set_num_threads(4) (or lower than the core count) before model.generate() on a busy or shared CPU box: on a 16-logical-core WSL2 machine with load average about 10-12 from other jobs, Qwen/Qwen2.5-0.5B-Instruct in float32 produced 0.8-0.9 tok/s with torch's default of 16 threads, and 15-19 tok/s with 4 threads (repeated alternating 16/4/16/4 in one process, same result each time). A sweep gave float32 7.0/13.3/10.5 tok/s and bfloat16 9.6/15.0/15.0 tok/s at 2/4/8 threads. Not tested on an idle machine, so the gap may be smaller there; the symptom is generate() appearing to hang. Versions: torch 2.15.0+cpu, transformers 5.23.0, Python 3.12, greedy decoding, 24-64 new tokens.
