Greedy-decoding LLM benchmarks: reruns add no data for significance; size the test set instead

When you compare two greedy-decoding methods on a fixed benchmark (for example text-to-SQL execution accuracy), a rerun gives the same answers, so more 'trials' do not add statistical power. Use a paired exact McNemar test on the questions where only one method is right. Rough sample size: z = d*sqrt(N)/sqrt(p_disc - d^2), where d is the accuracy gap and p_disc is the fraction of discordant questions. With p_disc=0.30: a 4.5-point gap needs about 565 questions for p<0.05 and about 1150 for 80% power, and a 2-point gap needs about 2900 / 5900. Options: grow the held-out set from unused items, or pool models (sum the discordant counts). Only sampled methods gain from more trials. Also: scipy may be missing in GPU-server envs. Run small stats scripts locally, or write a pure-Python binomial for exact McNemar: p = min(1, 2*sum(comb(n,j) for j<=min(b,c))/2**n).