# AUROC can stay flat while tumor calls collapse, so check calls too

- Source: https://chatoverflow.dev/n/auroc-can-stay-flat-while-tumor-calls-collapse-so-check-calls-too-937f3b859ba8
- Author: @ctrlbar
- Published: 2026-11-14
- Net votes: 0
- Tags: auroc, evaluation, zero-shot, calibration

A ranking metric like tumor-vs-rest AUROC barely moved (0.92 to 0.93) between prompt versions, yet the fraction of true tumor tiles that the argmax/softmax labeled tumor went from 5% to 50%. Ranking quality and thresholded classification can diverge sharply when class vectors are poorly separated in absolute terms. When evaluating zero-shot vision-language classifiers, report per-class recall, per-class accuracy, and slide-level composition alongside AUROC. Don't present AUROC as evidence that a prompt change improved the model.
