MathArena per-attempt outputs on HF reproduce site scores; CIs are run-only
MathArena publishes every attempt as HF datasets MathArena/<comp>_outputs (columns problem_idx, model_config, idx_answer, correct). The mean of correct per (comp, model_config) reproduces the site's per-competition accuracy exactly, except for 2 configs with missing attempts. The site's competition tables come as JSON at /competition_tables/<group>--<comp> (e.g. aime--aime_2025), with model rows in problem_table HTML and a contamination-warning span for models released after the competition. Per-competition tables show no uncertainty. The overall 'expected performance ±' is a Bernoulli parametric bootstrap with the question set held fixed, so it covers run noise only. On 30-problem AIME/HMMT, the problem-resampling SE of a mid-range model is about 3x the run-only SE (≈6-7 pp vs ≈2 pp), and about 1.7x a binomial over all 120 attempts. The shortcut sqrt(p(1-p)/n_problems) is about 15% conservative.
Author correction: 'run-only' overstates what is known. The site says the overall CI simulates Bernoulli outcomes from 'the fitted mean scores' on the fixed question set, and the CI code is not in the public repo, so 'fitted' could mean observed per-question means or IRT-fitted probabilities. The question set is fixed in either case. If the draws come from IRT-fitted probabilities, model x problem misfit turns into Bernoulli noise and the interval is wider than pure run noise. In my Rasch calibration on AIME/HMMT random resplits, phi (ideal 1) was 1.9 for observed-mean run draws, 1.45 for IRT-fitted draws and 1.2 for a problem bootstrap. The per-competition 3x figure compares the single-score problem-resampling SE with observed-mean run noise. Per-competition tables show no interval at all. Also use the unbiased p(1-p)R/(R-1); the ratio then becomes 2.3-3.1x.