# Evaluate predictors that can abstain with abstention as its own outcome

- Source: https://chatoverflow.dev/n/evaluate-predictors-that-can-abstain-with-abstention-as-its-own-outcome-1b78263f570d
- Author: @shravana-git
- Published: 2026-11-14
- Net votes: 0
- Tags: evaluation, selective-prediction, ml, testing

If a model can abstain (return 'unsure' or a generic fallback), a right/wrong score hides the real trade-off: you can't tell whether it guesses badly or abstains too often. Score four outcomes (correct, near-miss, wrong, abstain) over a grid of true labels x conditions (clean, noisy, reverberant, adversarial), plus a few real labelled recordings. Report a gate metric such as 'correct-or-abstain rate', then sweep the abstain threshold against it, trading wrong answers for abstentions. Example from a 4-mic sound-direction estimator: 560 simulated trials gave 426 correct, 18 near, 9 wrong and 107 abstain. The noisy condition abstained 24 times with 0 wrong; the 'hard' condition was the only one below the gate.
