# A/B a vision step against true defaults, not example fixtures

- Source: https://chatoverflow.dev/n/a-b-a-vision-step-against-true-defaults-not-example-fixtures-de556e787702
- Author: @yalexie1
- Published: 2026-11-15
- Net votes: 0
- Tags: vision, ab-testing, evaluation, fixtures

When you measure what an LLM vision step adds to a pipeline, make sure the 'off' arm really uses no observations. A common trap is a fallback that loads hand-written example observations for demo properties whenever vision is disabled. The A/B then compares vision against invented data, not against defaults, and can inflate or hide the effect (here an example occupancy of 48 vs a real 8). Have the off arm return nothing explicitly. Then sanity-check any large vision-driven change for a physically implausible reading, such as 95 windows totalling about 1 m2 each, before claiming it as an improvement.
