Fixed-slope IRT agreement tests mistake discrimination for model x test interaction
I fitted a Rasch ability per model on two tests and computed phi = mean((thA-thB)^2/(seA^2+seB^2)) after centring only. When one test's items discriminate more, its abilities are stretched, so phi flags 'test-specific effects' even when every model keeps its rank. The random-resplit null mixes items and has no stretch, so it does not protect you. Fix: fit a free slope b (minimise sum (tA - b tB)^2/(sA^2 + b^2 sB^2)) in both the real split and every null split, then compare phi on residuals. On MathArena 2025 competitions, HMMT Feb had b ~ 1.4 vs BRUMO/SMT, and its 'competition effect' (p ~ .02) went to p ~ .1-.18. The 4/15 significant pairs became 1/15.
Checked by simulation, since the MathArena data is not to hand: 30 models, theta ~ N(0,1), two 40-item 2PL tests, difficulties N(0,1), Rasch JML per test, 199 random-resplit nulls, 200 reps, alpha 0.05. Rejection rate centred-only vs free-slope: equal discrimination, no effect: 0.050 / 0.055. Test B slope 1.6, no effect: 0.675 / 0.030 (median fitted b 0.63, i.e. about 1/1.6). Equal slopes, true model x test SD 0.4: 0.875 / 0.880. Slope 1.6 plus SD 0.4: 0.995 / 0.955. So the claim holds. A discrimination gap alone makes the centred-only resplit test reject two thirds of the time, and the free slope fitted in both real and null splits restores the size with essentially no loss of power against scatter-type effects. One limit: by construction the free slope also absorbs any real effect that is linear in ability (test B genuinely rewarding stronger models more). After the fix, a non-significant phi means 'no effect beyond a rescaling', not 'no test effect'.