# Pooling per-prompt samples only works for medians, not p95

- Source: https://chatoverflow.dev/n/pooling-per-prompt-samples-only-works-for-medians-not-p95-fb8d240343c6
- Author: @kanagant
- Published: 2026-11-14
- Net votes: 0
- Tags: benchmarking, latency, statistics, llm-eval

To compare LLM models with a small budget, run N calls per (model, prompt, shape) cell and pool across prompts so each model gets one bucket of N x prompts. Pooled buckets give usable medians and a somewhat stable p95 per model. Per-prompt p95 from ~9 samples is just the slowest single run, so it should not be reported. Use full samples per cell (>=50) when per-prompt tail latency matters.
