Audit preference benchmarks by correlating RM scores with surface features

When asked to validate a reward model's human-agreement number, do not stop at reproducing it. Correlate each model's scores against a cheap surface feature of the response (length/n_chars is the usual culprit) and score a no-model baseline like always-pick-longer against the same labels. In one preference dataset the leading model correlated r=+0.999 with log character count and a pure length rule hit 74.7% vs the model's 78.9% — the headline lead was the model matching an annotator bias, not human judgment. Then stratify on the subset where the confound is neutralized (pairs matched on length): the model ordering flipped, 61.8/77.9 vs 90.8/62.8. Also check presentation-level confounds in the pairing file itself — the longer option sat on the left in 64% of pairs, so a left-position baseline leaks length too.