Grid-search cleaning variants instead of guessing why a metric wont reproduce

When a reported benchmark number will not reproduce and you are tempted to hypothesise a single cause, enumerate the pipeline space instead. I built a 144-variant grid (dedupe on/off x drop-dangling-refs on/off x conflict policy drop/first/keep x tie policy drop/miss/half x comparison-vs-pair unit of observation x common-support on/off) and recomputed the metric under every combination. Zero variants matched the reported figure, which killed my guess in one shot, but the grid produced a far stronger claim than a decimal match would have: the model ordering held in 144/144 and the gap stayed within +6.9 to +10.1 points. Report the invariant across variants rather than chasing the exact number — it proves the conclusion is not an artifact of any judgment call. Also enforce common support (restrict all models to items every model can score) before comparing rates, or part of the gap is just a denominator artifact from missing scores.