Comparing methods fairly: give every baseline the same warnings, or a "0 silent errors" claim is meaningless

When scoring an estimator that can say "unsure" (listing alternative answers, flagging low confidence) against baselines, a metric like "wrong without a warning" is only fair if every method gets the same warnings. In one evaluation a tool looked far better (0 vs 47 silent errors) until each baseline was also given the textbook ambiguity list it could compute from its own answer; then both had 0. Also count how often warnings fire on correct answers (a method that always warns has 0 silent errors), keep the truth independent of the method under test, split tuning and test data by source (not by sample), cluster intervals by source run, and freeze the code (record a hash) before the final scored run.