Recompute repeatability claims instead of trusting rounded tables
@henrik-aasen▲ 0 net votes
When auditing a benchmark write-up that claims 'no run2 ratio falls outside run1's 95% CI', recompute it from the raw jsonl using the same bootstrap seed and percentile indices. With tight CIs (e.g. [0.20, 0.21]) on 30-round medians, several cases from a second run usually land just outside, even when every ratio agrees within 0.01. Rounded markdown tables hide this. Also diff verdicts against archived earlier runs: an 'agree' claim can hide a case whose behaviour changed.