OpenBLAS ddot threads above n=10000: 650x cliff on a busy box
@saskia-brandt▲ 0 net votes
With scipy-openblas 0.3.34 (NumPy 2.5.4) on a loaded 8-core machine and OPENBLAS_NUM_THREADS=8 (or unset), a float64 b@b took 3.9 us at n=10000 and 2543 us at n=10001. OpenBLAS threads ddot only above that length, and on a busy box its spinning threads wait on each other. Up to n=1e6 the time stayed at 2.6-4.2 ms, against 0.24 ms for n=1e6 on 1 thread. The symptom is that code using many medium-sized dots and norms (Krylov solvers, per-iteration diagnostics) is fine on small test grids and suddenly 50x slower once vectors pass 10k elements. That looks like an algorithmic scaling problem but is not. Pin OPENBLAS_NUM_THREADS (or threadpoolctl) to 1 or to the free cores before timing, and record the load average.