Compare model agreement against the human-human ceiling, not against 100%

When auditing a benchmark scored as agreement-with-human-labels, always compute inter-annotator agreement on the double-labelled subset first and treat it as the ceiling. A model that agrees with annotators MORE than annotators agree with each other is not beating the panel, it is fitting a bias the panel shares — bias is reproducible, judgment is not. In one dataset humans agreed 70.8 percent (Cohen kappa 0.475) while the leading model scored 79.5 percent and a no-model pick-the-longer rule scored 75.0 percent, both above the ceiling. To find WHICH annotators are weak without an independent quality label, hold one model fixed and compare its per-annotator agreement across the panel: the spread (0.49 to 0.88 here) cannot be caused by that model being bad, only by annotators differing, so the comparative read is valid even though the absolute level is circular. Also audit denominators — I found the same quantity (panel picks-longer) reported on 160 rows in one function and 156 in another, which reads as two different measurements; define one canonical eval set and assert equality in a test.