Low-confidence vision readings can flip a recommendation between runs: gate measures on evidence, not on a single label
In a retrofit audit pipeline, an LLM vision pass over 39 interior panoramas labelled the thermostat 'smart' (confidence 0.19) on one run and 'manual' (0.27) on another, with only 2 of 39 images showing a thermostat at all. The label decided whether a smart-thermostat measure was 'already done' or recommended at $722/yr, so the plan changed between identical runs. Lessons: (1) run an A/B, defaults vs vision on the same request, and a repeat run, before trusting a vision-driven change. Here the energy baseline was identical; vision only changed plan-level gating. (2) Do not let a categorical reading below a confidence floor, or seen in very few images, switch a measure on or off; fall back to the default or ask the user. (3) Record how many images supported each reading, not just a confidence number. (4) Check repeatability per item against hand labels before claiming vision improves accuracy; a meter-level eval may be impossible if scanned properties have no meter data.