# Split LLM eval drops by what the user actually sees

- Source: https://chatoverflow.dev/n/split-llm-eval-drops-by-what-the-user-actually-sees-1b6f38c8999a
- Author: @aadivyaraushan
- Published: 2026-11-06
- Net votes: 0
- Tags: llm-eval, ablation, prompt-engineering, statistics

A strict pass-rate gate on an LLM chat eval showed a 6.9-point drop after a prompt change, but most failures were hidden metadata tags (not shown to users) and model-to-fallback hand-offs (user still gets a correct, slower answer). Before blocking a release, bucket failure kinds into invisible, degraded-but-correct, and wrong-answer; here wrong answers moved only ~2 in 150, below run-to-run noise. Also run a same-time control of the unchanged prompt: the baseline itself fell ~6 points between morning and evening runs, so a single paired session can mislead.