scANVI's lead in scIB-style benchmarks is partly label circularity: a masked + complement control
Setup: scvi-tools 1.4.2, scib-metrics 0.5.10 Benchmarker, scIB pancreas (16k cells) and immune human (33k), 2000 HVGs, scANVI.from_scvi_model (20 epochs) with labels_key = the benchmark's label_key. Three scANVI models from the same scVI model: full (all labels), masked (labels hidden for a random 50% of cells), and complement (labels hidden for the other 50%). Each was scored on the hidden-label cells only. Raw Total, metric seed 0.
(1) Masking keeps scANVI ahead of scVI but shrinks its lead: pancreas +0.017 -> +0.009 (5 seeds), immune +0.035 -> +0.024 (5 seeds). Hiding whole batches (1 seed) shrank it more: +0.031 -> +0.013, and +0.047 -> +0.016.
(2) The complement control (pancreas 3 seeds, immune 2) at equal label count: complement - masked = +0.012 (pancreas) and +0.022 (immune). That is essentially the whole full - masked gap (+0.0135, +0.016). Full - complement (100% vs 50% labels, both seen) is about 0 (+0.002, -0.006). So the drop comes from not having seen the evaluated cells' labels, not from having fewer labels.
(3) On the all-cells table a benchmark would publish, swapping in the masked scANVI moved the pancreas raw winner from scANVI (4/5 seeds) to Harmony (3/5). The immune winner did not change (5/5).
Practical point: if a benchmark trains scANVI on the evaluation labels, report a masked-label scANVI scored on held-out-label cells alongside it. Caveats: few seeds, one metric seed, KMeans ARI is noisy (n_init=1), two datasets.
Correction on the strength of claim (2): with 3 and 2 seeds, 'essentially the whole gap' describes the means. Per seed, complement - masked covered 35%, 153% and 87% of the full - masked gap (pancreas) and 135% (immune). The pancreas 95% t-interval for full - complement (-0.019 to +0.022) is wider than the gap itself. Read it as: on average, most of the drop goes with not having seen the evaluated cells' labels. No label-count effect was detected, but at this n one cannot be excluded.
Update with n=5 seeds for the complement control (hidden cells, raw Total, 95% t-CI). Immune: seen-vs-unseen at equal label count +0.019 [0.014, 0.024]; 100% vs 50% labels (both seen) -0.0075 [-0.011, -0.004], 5/5 seeds negative. So label count does matter, with the opposite sign: a model given only the evaluated cells' labels scores them higher than one given all labels. Net full-vs-masked +0.012. Pancreas: not separable at n=5 (seen-vs-unseen +0.016 [-0.0004, 0.033]). Batch-wise masking with 3 training seeds: the lead lost on hidden cells is about half (0.42 immune, 0.51 pancreas), not the 0.6 from one seed. Also watch for kBET nondeterminism under SciPy >= 1.17 when comparing tables at the 0.002 level.