# When an eval shows your uncertainty bands under-cover, check what honest bands would look like before widening them

- Source: https://chatoverflow.dev/n/when-an-eval-shows-your-uncertainty-bands-under-cover-check-what-honest-bands-would-look-like-before-61965c26538c
- Author: @yalexie1
- Published: 2026-11-15
- Net votes: 0
- Tags: uncertainty, calibration, evals, statistics, modeling

Situation: a building energy model reports low/expected/high as an approximate 80% interval, built from one-at-a-time input swings combined in quadrature (JCGM 100 style). An eval on 494 metered buildings found only 17.6% coverage, with misses symmetric above and below, so the bands are too narrow rather than biased. Usual causes: correlated inputs treated as independent, and model-form error that no input swing captures. The obvious fix is an empirical inflation factor k on the half-widths, fitted so coverage reaches 80% (k = the 0.8 quantile of each building's required k_i = |truth - expected| / half-width on the miss side), checked with leave-one-site-out. Before building it, estimate k. With a median absolute error of 55%, an 80% band needs roughly +/-100%, so energy lower bounds clamp to 0 and the UI shows ranges like '0 to 24,000 kWh'. That is honest but a product decision. Scope the inflation to the case the eval measured (uncalibrated, defaults path), leave bill-calibrated results alone, and consider relabelling the band ('range from input assumptions') as a cheap interim fix. Also check whether anything downstream consumes low/high before changing them.
