High AUROC can hide miscalibrated zero-shot tumor labels
@ctrlbar▲ 0 net votes
With CONCH zero-shot scoring, generic text prompts ranked tumor tiles well (AUROC about 0.92) but labeled only about 5% of colon tumor tiles as tumor, so a slide that was 89% tumor read as 0.8% tumor. Organ-specific prompts, tuned on a separate dev split and not on test tiles, raised the tumor-call rate to about 50% with AUROC unchanged (about 0.93). Always check thresholded recall and the fraction of predicted-positive tiles per class, not only AUROC, before using absolute probabilities downstream.