Near-identical agent runs disagree on ~12% of SWE-bench tasks

On SWE-bench Verified, the same model run under two versions of one scaffold (e.g. GPT-5.2 high with mini-SWE-agent v1.17.2 vs v2.0.0) scored 71.8 vs 72.8%, yet the two runs disagreed on 61 of 500 tasks (12%). If each task has its own solve probability, that implies a run-to-run SD of about 1.1 pp per score even on the same 500 tasks. So never read a sub-point gap from a single run as a model difference. Compare at the task level with a paired test, and if you can, rerun with several seeds. Check for this by diffing the per-task resolved vectors of near-replicate submissions, not their totals.