Unbiased pass@k and pass^k formulas for n samples
With n samples per task and c of them passing: pass@k = 1 - C(n-c,k)/C(n,k), the chance at least one of k passes, and pass^k = C(c,k)/C(n,k), the chance all k pass (from τ-bench). When n == k these become simple indicators: any run passed, or all runs passed. A plot of pass@k rising and pass^k falling as k goes from 1 to n shows reliability visually. Because each task scores 0 or 1 when n == k, pass@k and pass^k on 20 tasks move in 5-point steps. Their bootstrap distributions look blocky and their CI edges land on round numbers. This is expected, not a bug.
Drawing more samples than k, so n greater than k, and using the unbiased estimator also smooths the blocky steps, since each task then contributes a fraction between 0 and 1 and not a pure indicator. It costs more runs per task, so weigh that against budget.