Chatbot Arena 95% CIs cover the 3-month-later rating only ~79%
I checked how often a model's published Arena interval (rating_lower..rating_upper in lmarena-ai/leaderboard-dataset, text/overall) contains the same model's rating ~90 days later. Board-wide shifts were removed by subtracting the median change of established anchor models. One pair per model, 240 models, 2024-2026. Coverage was 78.7% [73, 84]. If only vote sampling moved ratings, it would be ~99%, because the later estimate pools the earlier votes. Coverage is ~95% when few new votes arrive and falls to 74% when new votes are half or more of the total. Misses are symmetric (13% below, 8% above), so it is diffusion, not aging. Intervals ~1.84x wider (±10.6 vs ±5.8 points) would cover 95%. Takeaway for anyone using Arena CIs: they quantify vote resampling, not non-stationarity of the voter or prompt stream (or silent model updates). Don't treat them as the uncertainty of a rating's future value.
Refinement after an independent review: part of the shortfall comes from board-wide refits, not new votes. Five board-to-board steps reshuffle established models non-uniformly (robust spread > 1 point; three large ones are recounts/refits on 2024-06-26, 2025-07-25 and 2025-09-17), and median-anchor correction cannot remove that. Excluding pairs whose window spans any of them: 85.2% [79, 91] coverage vs 99.1% expected (169 models). So refits explain about a third of the gap and the rest persists. Clustered bootstraps (by month or window) widen the CI only slightly. Also: the inflation that reaches the expected coverage is 2.6x; 1.84x is the factor for 95% coverage.