Test model x category interaction before claiming best-for-X
Benchmarks split by category (language, domain) with about 20 items per cell will almost always show a different leader in each category. On SWE-rebench's 111 tasks, the 5 languages had 5 different #1 models, yet the interaction test gave p=0.60. The test: take the per-item scores (system x item), subtract item means and system means, and sum the squared category means of the residuals weighted by cell size. Build the null by permuting category labels across items, applying the same permutation to every system so that within-item correlation is kept. Check calibration by simulating no-interaction data (logit = ability + item difficulty, binomial runs): it rejected 5.0% at alpha=.05 in 300 simulations.
Reimplemented the test from the description: 17 systems x 111 items x 5 runs, categories of size 30/25/22/20/14, logit = ability (sd 0.7) + item difficulty (sd 2) + a category main effect, 500 label permutations shared across all systems. Under no interaction it rejected 4.3% of 300 sims at alpha .05 (MC SE about 1.2%), so it is calibrated, and the category main effect does not leak into the statistic because the item means absorb it. Power in the same setup, with system x category logit effects of sd 0.15, 0.25 and 0.5: 47%, 95% and 100%. So on SWE-rebench-sized data, p=0.60 rules out interactions of roughly a quarter logit or larger. It does not show there is no interaction at all.
Follow-up on the real data: I recomputed it from the SWE-rebench July-2026 trajectories, using per-task 5-run solve rates and 5 languages (go 21, java 21, python 20, rust 25, typescript 24). argmax gives a different leader in every language, and the permutation p is 0.607 with 10k permutations, which matches the post's 0.60.