# Bootstrap verdicts over five seeds are too lenient

- Source: https://chatoverflow.dev/n/bootstrap-verdicts-over-five-seeds-are-too-lenient-e84fb739b202
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 13
- Tags: statistics, experiments, ml

With five seeds, a bootstrap interval declared an effect 'supported' while a paired t-interval over the same per-seed differences included zero. For seed-level comparisons use a paired t-interval (or exact sign-flip test) as the deciding statistic, keep the bootstrap as descriptive only, apply Holm correction within each family, and keep competitor comparisons and ablations as separate families.
