Parametric bootstrap from fitted IRT probabilities is not run-only noise
To get a run-noise-only SE for an IRT/Rasch ability, you might redraw each (model, problem) cell as Binomial(R, fitted p). In a simulation from a pure Rasch model this looks perfectly calibrated, because there fitted p equals the true cell p. On real LLM eval data, cells are close to deterministic (a model solves a problem 4/4 or 0/4 far more often than Rasch predicts), so drawing from fitted p turns model x problem interaction into fake run noise and widens the 'run-only' SE. For pure run noise, redraw from the observed cell means and scale the SE by sqrt(R/(R-1)), since the plug-in p(1-p) is biased low by (R-1)/R. More generally, a known-truth simulation that leaves out the interaction term cannot validate an estimator whose behaviour depends on that term. Check what the simulation omits before accepting a fix it 'validates'.
Confirmed by simulation, with one simplification: I bootstrapped each model's mean score, not a Rasch ability. Setup: 12 models x 300 problems, R=4, true cell p = expit(theta_m - b_p + gamma_mp) with theta ~ N(0,1), b ~ N(0,2), gamma ~ N(0, s). Truth is the run-only SE given the cell p. The 'fitted' bootstrap draws Binomial(R, p) from an additive-logit (Rasch) binomial ML fit. 400 bootstrap reps, 3 seeds. Median ratio of bootstrap SE to true run-only SE:
s=0: fitted-p 0.98-1.02, observed means x sqrt(R/(R-1)) 0.98-1.01 s=1: fitted-p 1.06-1.11, observed 0.98-1.03 s=3: fitted-p 1.39-1.42, observed 0.99-1.01 (63-66% of cells at 0/4 or 4/4)
So the pure-Rasch simulation (s=0) does make the fitted-p bootstrap look calibrated, and it overstates run noise by about 40% once the interaction is large. The observed-mean bootstrap with the R/(R-1) correction is on target in all three settings.