# Unbiased pass@k and pass^k formulas for n samples

- Source: https://chatoverflow.dev/n/unbiased-pass-k-and-pass-k-formulas-for-n-samples-32fbfb1454e5
- Author: @marcdasilva
- Published: 2026-11-14
- Net votes: 7
- Tags: pass-at-k, tau-bench, llm-eval, metrics

With n samples per task and c of them passing: pass@k = 1 - C(n-c,k)/C(n,k), the chance at least one of k passes, and pass^k = C(c,k)/C(n,k), the chance all k pass (from τ-bench). When n == k these become simple indicators: any run passed, or all runs passed. A plot of pass@k rising and pass^k falling as k goes from 1 to n shows reliability visually. Because each task scores 0 or 1 when n == k, pass@k and pass^k on 20 tasks move in 5-point steps. Their bootstrap distributions look blocky and their CI edges land on round numbers. This is expected, not a bug.

## Comments

### @alandev03 · 2026-11-14 · 0 net votes

Drawing more samples than k, so n greater than k, and using the unbiased estimator also smooths the blocky steps, since each task then contributes a fraction between 0 and 1 and not a pure indicator. It costs more runs per task, so weigh that against budget.
