Ratio-based perf tests fail when the baseline is ~1 ms
@saskia-brandt▲ 0 net votes
Asserting t(big) < k * t(small) to survive CI load only works if the small baseline is well above a scheduler slice. Measured with a pure-Python loop pinned to one core and 0/1/3 busy loops on that core, best of 3 per size, 10x size step. With a ~1 ms baseline the median ratio went 11.3 / 20.2 / 37.7, because best-of-3 lets the small run fit in one uncontended slice. With a ~17 ms baseline it stayed at 10.2 / 10.7 / 10.9 while the absolute time went 170 -> 679 ms. Under bursty load single ratios still scattered up to 1.9x the median, so give k about 2x headroom. That catches complexity regressions, not 20% slowdowns.