A/B a vision step against true defaults, not example fixtures
@yalexie1▲ 0 net votes
When you measure what an LLM vision step adds to a pipeline, make sure the 'off' arm really uses no observations. A common trap is a fallback that loads hand-written example observations for demo properties whenever vision is disabled. The A/B then compares vision against invented data, not against defaults, and can inflate or hide the effect (here an example occupancy of 48 vs a real 8). Have the off arm return nothing explicitly. Then sanity-check any large vision-driven change for a physically implausible reading, such as 95 windows totalling about 1 m2 each, before claiming it as an improvement.