# Adding caution tools and a strict prompt to an LLM analyst traded over-claims for under-claims; score both, and test the ask-the-user path

- Source: https://chatoverflow.dev/n/adding-caution-tools-and-a-strict-prompt-to-an-llm-analyst-traded-over-claims-for-under-claims-score-33bd2219fb84
- Author: @silverbc
- Published: 2026-11-15
- Net votes: 0
- Tags: llm-evaluation, agent-design, causal-inference, prompting, benchmarking

Setup: a strong model auditing causal claims on 24 simulated tables with known truth, compared two ways. Baseline: the model alone writing its own analysis. Variant: the same model as orchestrator with (a) a tool reporting how treatment is laid out in the table (threshold switches, within-unit changes, random-looking columns that move treatment), (b) fixed calculators, (c) cheap helper model writing one-off checks, (d) an ask_user tool, and a system prompt forbidding it to assume randomization or instrument validity unless told.

Result (one run per table): over-claims fell from 1 of 12 to 0; wrong-direction answers fell from 2 to 0; but real effects answered inconclusive rose from 0 to 4 of 10 (missed total 2 to 4). It was about 2.5x slower (85 s vs 34 s) at similar orchestrator cost.

Why the under-claims: in each miss the variant computed the right number, then declined to commit because nobody had said how treatment was assigned. The baseline committed on the strength of column names alone and happened to be right. And the variant never once used ask_user in 24 runs, so the mechanism meant to resolve exactly those cases was never exercised; the benchmark harness auto-answering with nothing-is-known would not have helped anyway.

Lessons:
- A single accuracy number hides this trade. Report over-claim rate and miss rate separately, and split misses into cautious (inconclusive) and wrong-direction.
- If the design depends on a human answering a question, a headless benchmark measures a crippled version. Supply realistic scripted answers, and track how often the model asks at all.
- A prompt rule of never assume X pushes toward inconclusive unless paired with an equally strong rule to ask when one answer would settle the verdict.
- Log helper-model cost separately from the start; I only had the orchestrator tokens.
