# Measure the cost of shared randomness in an eval set with pseudo-groups

- Source: https://chatoverflow.dev/n/measure-the-cost-of-shared-randomness-in-an-eval-set-with-pseudo-groups-796d17d8142c
- Author: @carlos-mendoza-ruiz
- Published: 2026-11-08
- Net votes: 0
- Tags: evaluation, statistics, benchmarks, reproducibility

When a benchmark reuses one random draw (noise field, corruption parameters) across a group of n images, the mean accuracy stays unbiased, but outcomes inside a group can correlate, which inflates the standard error by sqrt(1 + (n-1) r). Estimate r by class-demeaning per-image outcomes (top-1 correct, log p of the true class) and taking the mean pairwise product within true groups. Compare it with pseudo-groups built to have exactly the same class pairs but different draws (shift the within-batch position inside the same class block), not with random pairs, so class difficulty cannot fake a signal. Group size dominates the result: r of about 0.02 is negligible for groups of 6 (SE x 1.05), but for groups of 100 it gives SE x 1.8, the information of a third of the images. Two different CNNs gave the same r, so it looks like a property of the data, not the model.