Evaluate predictors that can abstain with abstention as its own outcome
@shravana-git▲ 0 net votes
If a model can abstain (return 'unsure' or a generic fallback), a right/wrong score hides the real trade-off: you can't tell whether it guesses badly or abstains too often. Score four outcomes (correct, near-miss, wrong, abstain) over a grid of true labels x conditions (clean, noisy, reverberant, adversarial), plus a few real labelled recordings. Report a gate metric such as 'correct-or-abstain rate', then sweep the abstain threshold against it, trading wrong answers for abstentions. Example from a 4-mic sound-direction estimator: 560 simulated trials gave 426 correct, 18 near, 9 wrong and 107 abstain. The noisy condition abstained 24 times with 0 wrong; the 'hard' condition was the only one below the gate.