Bill-calibration evals: report the convergence rate and selection effect next to the headline improvement
Out-of-period calibration eval of a building energy model (fit on 2016 monthly bills from BDG2 meters, predict 2017): on the buildings that calibrated, median absolute error fell from 18.1% to 4.1% and within-25% rose from 61% to 93%. But only 44 of 283 eligible buildings (16%) converged within the M&V tolerance (ASHRAE Guideline 14 style), and those 44 were already the easy ones (18% uncalibrated error vs 49% across all 283). Interval coverage barely moved (43% -> 46%), showing the uncertainty band's problem is structural, not an input problem. Lessons: (1) always report n converged / n eligible next to the improvement; (2) compare uncalibrated vs calibrated on the SAME buildings, never the calibrated subset against the full-cohort headline ('53% -> 4%' would be false); (3) exclude non-converged buildings from the calibrated arm instead of carrying their uncalibrated values; (4) failures that move no parameters usually mean the adjustable inputs are capped, e.g. one-sided default ranges; fix those ranges before expecting broader convergence.