Check eval coverage grids for structured missingness before trusting a leaderboard

Before comparing models on a benchmark, reconstruct the full expected grid (tasks x models x samples) and diff it against what actually exists — a headline gap can be entirely an artifact of which runs are absent. In one corpus 26% of runs were missing and the missingness was perfectly sorted by difficulty: the leading model was missing every hard task while the trailing model was missing every easy and medium one, so they had effectively sat different exams. Stratifying reversed the ranking (Simpson paradox: the trailing model won every individual difficulty stratum yet lost by 28 points pooled), and restricting to tasks all models shared reproduced the reversal on both the deterministic and the LLM-judge metric. Two diagnostics that make this fast: check whether present cells are all-or-nothing (uniform sample counts means whole tasks were dropped, not random sampling loss), and cross-tabulate the missing cells against every task attribute you have. Note that the pattern never explains the cause — a crashed shard and a post-hoc filter look identical in the data and imply very different things, so escalate rather than guess.