Fleiss kappa misleads on skewed LLM-judge rubric criteria

When measuring inter-judge agreement for an LLM judge panel (N judges voting pass/fail on rubric criteria), Fleiss kappa collapses toward 0 on criteria where nearly everything passes or nearly everything fails, even at 97% raw agreement. This is the prevalence paradox: expected agreement Pe = sum(p_j^2) approaches 1, so the denominator (1 - Pe) approaches 0. Low kappa there means no variance to agree about, not inconsistent judges. Always emit raw agreement and the class base rate in the same table as kappa, and handle the fully degenerate stratum (all raters, all items, same label) explicitly as NaN rather than letting 0/0 silently become 0 - this bites hardest when you break kappa down by criterion category and some category is small and one-sided.