YOLOv8n ONNX on CPU with onnxruntime: about 10 FPS at 640x640 (93-98 ms/frame); on a busy host intra_op_num_threads=1 was faster than all cores
Expect roughly 10 FPS for yolov8n.onnx (640x640, opset 18, onnxruntime 1.32.0, CPUExecutionProvider) on a laptop-class x86 CPU when other work is running. If the machine is loaded, set so.intra_op_num_threads explicitly and measure; the default thread pool can be slower than one thread.
Numbers from one AMD Ryzen 7 7840HS (16 threads) under load average about 9:
- full pipeline (letterbox, BGR->RGB, inference, decode, NMS) over 100 frames of a synthetic 640x480 video: 93.1 ms/frame, 10.17 FPS end to end, 10.74 FPS inference only;
- inference only, 40 runs after 5 warm-ups: intra_op_num_threads=1 gave 97.7 ms (10.2 FPS); intra_op_num_threads=16 gave 128.7 ms (7.8 FPS).
With other processes on every core, onnxruntime's worker threads likely spin and wait on busy cores. On an idle machine more threads should win. Treat these as upper-bound latencies for a busy dev box, not a benchmark. A Raspberry Pi will likely be several times slower, so plan on 320x320 input, frame skipping, or a smaller or quantized model for real-time use there.
Snippet:
so = ort.SessionOptions(); so.intra_op_num_threads = 1
sess = ort.InferenceSession("yolov8n.onnx", so, providers=["CPUExecutionProvider"])Warm up a few runs before timing; the first inference is noticeably slower.