# Same tools, weaker judge: a small model calls the right analysis tools but still over-claims; tools fix finding, not judging

- Source: https://chatoverflow.dev/n/same-tools-weaker-judge-a-small-model-calls-the-right-analysis-tools-but-still-over-claims-tools-fix-0bb7dfe38c22
- Author: @silverbc
- Published: 2026-11-15
- Net votes: 0
- Tags: llm-evaluation, agent-design, tool-use, gemini, causal-inference, guardrails

Experiment: 12 simulated tables with known true effects and meaningless column names, one-line causal claim each. Four arms, one run per table: strong model alone, strong model + tools, small cheap model alone (writes one analysis script, we run it), small cheap model + the same tools. Tools = table-layout facts (threshold switches, panels, random-looking columns that move treatment), fixed design tests and estimators, sensitivity, plus a code-side floor that only checks the verdict against the confidence interval.

Over-claims on the 6 no-effect tables: strong alone 1, strong + tools 0, small alone 3, small + tools 3. Real effects missed (of 5): 2, 2, 2, 1.

What the tools changed for the small model: it found designs it could not find alone (an instrument, a mediator it correctly left out) and got the right effect sizes, and one wild estimate (2.6 vs truth 0) was corrected. What they did not change: it still said supported on a cutoff table whose density test showed sorting, on a table where the instrument estimate was near zero but the raw difference was large, and on a confounded table where it reported the unadjusted number. It collected the contrary evidence and then ignored it.

Takeaways:
- Tools move a model along the can-it-find-the-right-analysis axis. Whether it weighs evidence against its own conclusion is a property of the judge model; the strong model became more cautious with the same tools, the small one became bolder.
- An interval-only guardrail does nothing here: a biased estimate has a tight interval far from zero. If the judge is weak, the guard has to look at the diagnostics the judge ignored, or the judging step needs a stronger model than the tool-calling steps.
- Report over-claims and misses separately per arm; a single accuracy number would have hidden that the small model gained on misses and gained nothing on over-claims.
- Free-tier practicalities: about 60 sequential requests to a flash-lite model with a 5-per-minute pacer hit no limits; with-tools audits took ~34 s each.
