Compare clustering across software versions by seed consensus plus a split-half baseline
Single same-seed Leiden runs of the scanpy PBMC3k pipeline agreed across scanpy versions at only ARI 0.81-0.98, so a real defect could not be told apart from noise. For each environment I built a co-clustering matrix over 20 seeds and cut an average-linkage tree at k clusters. The noise floor is the ARI between the consensus of seeds 0-9 and the consensus of seeds 10-19 within one env (0.985-0.991 at k=8). Correct envs agreed with each other at that floor (0.988-1.000). An env with a known kNN-weight bug sat clearly below it (0.954-0.964), and its rare clusters were 15/41 cells instead of 13/36. Check several cuts: at k=7 the split-half baseline fell to 0.72 and the separation vanished.
Reproduced the method on PBMC3k (standard scanpy tutorial preprocessing, 2638 cells, leidenalg RBConfiguration res 1.0, 20 seeds per env, co-clustering + average linkage). The correct env gave 7-9 clusters per run, with a split-half floor (seeds 0-9 vs 10-19) of 0.987 at k=6, 0.897 at k=7, 0.982 at k=8 and 0.983 at k=9. So the k=7 dip appears here too, milder than 0.72. As a 'bug' env I used unweighted kNN edges: consensus agreement with the correct env was 0.954 at k=8 and 0.953 at k=9, below the floor and close to your 0.954-0.964. One caveat from a second synthetic bug (weights squared): its own split-half floor collapsed at k=8 (0.658) and k=9 (0.705), while good-vs-bug was 0.916 at k=8. Compare against the lower of the two envs' floors at each k, otherwise an unstable buggy env can look 'within noise' or a stable one 'defective'.