Split agent benchmark variance into run noise and item difficulty
If an agent was run k>=5 times per item, estimate each item's solve probability p_i. The binomial variance then splits exactly: pbar(1-pbar) = E[p_i(1-p_i)] (run noise) + Var(p_i) (item difficulty). Estimate E[p(1-p)] without bias as phat(1-phat)*k/(k-1) and bootstrap over items for a CI. On nebius/SWE-agent-trajectories (Llama 8B/70B/405B, median 14 runs per instance for 70B), run noise was 39-53% of the variance, about 1 pp SD of a 500-item score from rerunning alone. So 3 seeds shrink the SE by only about 14% (sqrt((1-s)+s/k)); more items help more. Download only the 4 needed columns via pyarrow; the parquet shards are about 1.1 GB in total. Check for duplicated rows by hashing trajectories: 15% of patches repeat, but no trajectories do.
Checked the maths by simulation (500 items, p_i ~ Beta(0.4, 0.9), k=14 runs, 200 replicates). phat(1-phat)*k/(k-1) averages 0.1203 against a true E[p(1-p)] of 0.1204, so the estimator is unbiased as stated; the decomposition is just the law of total variance for a Bernoulli. One number is optimistic: with run-noise share s from 0.39 to 0.53, sqrt((1-s)+s/3) is 0.860 to 0.804. Three seeds therefore cut the SE by 14-20%, and 14% is only the low end. Also say which SE you mean. The formula treats the items as a sample from a larger population. If the question is whether a re-evaluation on the same fixed 500 items would rank models the same way, only the run-noise term is left, and k seeds cut that SE by the full 1/sqrt(k). I did not re-download the trajectories, so the 39-53% figure itself is unchecked.