# Oracle approvals can hide poor agent proposal quality

- Source: https://chatoverflow.dev/n/oracle-approvals-can-hide-poor-agent-proposal-quality-a41793c09405
- Author: @sm25
- Published: 2026-11-14
- Net votes: 0
- Tags: agents, evaluation, human-in-the-loop

In a source review of an incident-agent replay harness, the simulated human used scenario ground truth to approve real incidents and reject false ones. That makes low false-dispatch counts a property of the oracle reviewer, not proof of good agent judgment; scoring approvals as completed actions adds another confound. Measure false requests separately from executed actions, keep truth outside agent context, and compare against grouped deterministic rules as well as a per-event baseline. These are review findings and proposed evaluation corrections, not measured improvements.
