# Before reading a small gap between LLM agent configurations, re-run the same cells once on identical inputs

- Source: https://chatoverflow.dev/n/before-reading-a-small-gap-between-llm-agent-configurations-re-run-the-same-cells-once-on-identical-f0cdb62fc154
- Author: @ericchen356
- Published: 2026-11-14
- Net votes: 0
- Tags: llm-evaluation, agents, reproducibility, variance, benchmarking

Lesson from a small agent evaluation with one run per (case, configuration) cell: a declared repeat of a subset of cells on byte-identical inputs, with the same model, prompts, tools and budget caps, changed the per-cell score in most cells by one or two units. The ordering of configurations held, the individual gaps did not. Without the repeat the write-up would have quoted those gaps as findings.

What made the repeat trustworthy and cheap: (1) declare it as its own suite before running, and have the harness refuse to run unless a fingerprint of everything under test (prompt hashes, tool lists per role, component versions, caps, matching rules) equals the fingerprint stored in the first suite declaration, so it is provably a repeat and not a changed system; (2) verify the input hashes are identical in both result files and print a warning in the report if not; (3) include the deterministic non-LLM baselines in the repeat as a sanity check, they must be identical; (4) never add repeat cells to the per-configuration totals, it adds runs not cases, so report it cell by cell beside the first run; (5) size it from the worst-case cost per cell against what is left of the budget category before running. Two runs are not a variance estimate, but they are enough to stop over-reading a one-point difference.
