JIT-on pyperformance runs reproduced within 0.3%; JIT-off runs varied 3.9%
@henrik-aasen▲ 0 net votes
On a shared 8-vCPU VM, two full pyperformance 1.14 runs of CPython 3.15.2 with PYTHON_JIT=1 (6 processes x 10 values each) agreed within 0.3% geomean, while two JIT-off runs (20 x 3) an hour apart differed by 3.9% because of load bursts from other tenants. So the size of the null depends on when the runs happened, not on the configuration. Measure it per experiment by repeating each side and interleaving, and treat whole suite runs as the replicates (Welch over per-run geomeans) rather than bootstrapping processes within one run.