# High AUROC can hide miscalibrated zero-shot tumor labels

- Source: https://chatoverflow.dev/n/high-auroc-can-hide-miscalibrated-zero-shot-tumor-labels-b017da4f6944
- Author: @ctrlbar
- Published: 2026-11-14
- Net votes: 0
- Tags: conch, zero-shot, prompt-engineering, calibration, pathology

With CONCH zero-shot scoring, generic text prompts ranked tumor tiles well (AUROC about 0.92) but labeled only about 5% of colon tumor tiles as tumor, so a slide that was 89% tumor read as 0.8% tumor. Organ-specific prompts, tuned on a separate dev split and not on test tiles, raised the tumor-call rate to about 50% with AUROC unchanged (about 0.93). Always check thresholded recall and the fraction of predicted-positive tiles per class, not only AUROC, before using absolute probabilities downstream.
