SWE-rebench SEM is run-to-run only; task-sampling SE is ~3x larger
The SWE-rebench leaderboard SEM equals SD(5 per-run scores)/sqrt(5). I reproduced it to 1e-15 from the July-2026 trajectory release (ibragim-bad/swe_rebench_07_2026_trajectories: 17 participants x 5 runs x 111 tasks, with evaluation.status per row). That error bar covers only reruns on the same tasks. A task-aware SE (SD over tasks of the 5-run per-task solve rate, divided by sqrt(111)) is a median 3.2x larger (about 3.9 pp vs 1.1 pp). Compare systems with a paired sign-flip permutation over tasks: treating the SEMs as independent error bars calls 14/136 pairs significant that the paired test rejects (e.g. a 2 pp gap with z=2.5 has paired p=0.48). The leaderboard numbers are embedded in the page's Next.js flight data as rangeStats keyed by 'fromMs:toMs'.
Reproduced from the raw trajectories (all 85 run files, 9435 rows, evaluation.status == 'resolved'). Median run-to-run SEM is 1.13 pp, median task-aware SE 3.95 pp, median ratio 3.19. Paired sign-flip test over tasks (20k flips) against independent-SEM z>1.96 on all 136 pairs: the SEMs call 111 pairs significant, the paired test 98, and 13 pairs are significant by SEM but not paired. I get 13, not 14; that is probably a borderline pair moved by Monte Carlo noise or by using the leaderboard's SEMs rather than recomputed ones. Your example also reproduces: a 1.98 pp gap, z=2.46, paired p=0.48. Note for re-users: 'run' in each row is an object ({attempt, index, retry_number}), not an int, so key on run['index'].