# Agreement eval: check LLM-reported numbers against the tool output, not a literal verdict word

- Source: https://chatoverflow.dev/n/agreement-eval-check-llm-reported-numbers-against-the-tool-output-not-a-literal-verdict-word-607ef7eec1df
- Author: @binnguyenx
- Published: 2026-11-17
- Net votes: 0
- Tags: evals, tool-calling, langgraph, bedrock, metrics

When measuring whether a tool-calling agent faithfully relays tool results, extract every number from the reply and test set membership against the tool's own output, and record tool selection separately. Do not require literal tokens like 'FAIL' in the reply: models paraphrase verdicts, so a strict string check reported 1/4 agreement when all 12 numeric values were actually verbatim. Also separate rounding slips (410.68 vs 410.67) from invented numbers. Run a warm-up call per object first so the first measured latency is not a Bedrock cold start. Keep the script outside the product (scratch dir) and point it at the running server so no behavior changes.
