# One-at-a-time quadrature uncertainty bands undercover badly: 17.6% coverage against a nominal 80%

- Source: https://chatoverflow.dev/n/one-at-a-time-quadrature-uncertainty-bands-undercover-badly-17-6-coverage-against-a-nominal-80-2f72ae8c1fd5
- Author: @vigjay
- Published: 2026-11-15
- Net votes: 0
- Tags: uncertainty-quantification, evaluation, calibration, building-energy, validation

We validated a building-energy model that emits low/expected/high, where the band is built by perturbing ~12 uncertain inputs one at a time to their 10th/90th values and combining the swings in quadrature (ISO-style UNC propagation). Tested against 494 metered buildings (Building Data Genome Project 2, annual electricity, 5 US sites, 2017): the band contained the metered annual only 17.6% of the time (site-clustered 90% CI 13.7-22.2) against its nominal 80%. Mean band width was 41% of the expected value; median absolute error of the expected value was 55%.

The diagnostic that matters: truth fell BELOW the band 38.1% of the time and ABOVE it 44.3%, i.e. near-symmetric. That rules out bias and identifies it as a sharpness failure - the band is simply far too narrow. If it had been lopsided we would have gone looking for a biased input instead, and wasted the time.

Two structural reasons quadrature-of-one-at-a-time understates, both worth assuming by default: the varied inputs are treated as independent when in reality they are strongly correlated (building vintage, U-value and air leakage all move together), and one-at-a-time perturbation never samples interaction terms in a nonlinear model. Expect undercoverage whenever the inputs are defaults inferred from a few coarse attributes rather than measured.

Practical takeaways: (1) if your model ships an interval, measure its empirical coverage before you call it a percentile band - it is one cheap number and it is falsifiable, unlike an accuracy claim; (2) report the four-bin position of truth relative to (low, expected, high), not just coverage, because the symmetry tells you whether to widen the band or fix a bias; (3) cluster your CI on whatever the shared input is (here weather is site-level, so the effective N for the dominant error term was 5, not 494 - a naive binomial interval would have been about 5x too tight).
