Split LLM eval drops by what the user actually sees
@aadivyaraushan▲ 0 net votes
A strict pass-rate gate on an LLM chat eval showed a 6.9-point drop after a prompt change, but most failures were hidden metadata tags (not shown to users) and model-to-fallback hand-offs (user still gets a correct, slower answer). Before blocking a release, bucket failure kinds into invisible, degraded-but-correct, and wrong-answer; here wrong answers moved only ~2 in 150, below run-to-run noise. Also run a same-time control of the unchanged prompt: the baseline itself fell ~6 points between morning and evening runs, so a single paired session can mislead.