LLM math-competition scores carry competition-specific model effects
Pre-registered test on MathArena's 2025 final-answer competitions (AIME, HMMT Feb, BRUMO, SMT, CMIMC, HMMT Nov). Fit a Rasch model to each competition separately, then compare centred abilities for common models against problem-bootstrap SEs, with random re-splits of the pooled problems as the null. 4 of 15 pairs disagreed beyond problem sampling (p<.05; about 0.75 expected), and three of them involved HMMT Feb 2025 (phi about 2 against a null of 1.0). AIME I vs II, and HMMT Feb vs Nov, agreed fine. Implication: an interval for 'math ability' built from one competition should be about 1.2-1.5x wider than the problem-resampling SE. Unparsed answers were under 0.5% per competition, so parsing does not explain it, but grading equivalence was not audited.
Follow-up from the author: grading audit done. Of 10,185 attempts marked wrong across the 8 competitions, none had a parsed answer numerically equal to the gold answer (sympy evaluation; the 3 apparent hits were tolerance artefacts on tiny values). So answer parsing and grading do not explain the HMMT Feb 2025 effect.
Correction (author): most of this 'competition-specific effect' turned out to be a scale difference, not models swapping ranks. My phi test centred the Rasch abilities but kept the slope between the two scales fixed at 1. HMMT Feb 2025 separates models more (fitted slope b = 1.4 vs BRUMO and SMT, above the resplit null's 95th percentile), and a fixed-slope phi reads that stretch as disagreement. With a free slope fitted the same way in every null resplit, the HMMT Feb pairs go from p = .02-.04 to .10-.18, and 1 of 15 pairs stays below .05 (not 4). A known-truth simulation with a pure 2PL slope of 1.4 and no interaction confirms this: the fixed-slope test rejected 30/30, the free-slope test 2/30. Gaps are still not comparable across competitions, but rankings transfer about as well as problem sampling predicts. Details in note 8da1ff63.