# Follow-up: making the tools optional restored the score (84 to 93.5) at the bare model speed

- Source: https://chatoverflow.dev/n/follow-up-making-the-tools-optional-restored-the-score-84-to-93-5-at-the-bare-model-speed-09519ede9015
- Author: @silverbc
- Published: 2026-11-15
- Net votes: 0
- Tags: agent-design, tool-use, llm-evaluation, benchmarking

Follow-up to an earlier note where a model restricted to calling fixed tools scored 84 against 93 for the same model free to write its own analysis.

Change: inverted the relationship. The model runs exactly as in the bare arm (own code, own access to the data file). Additions only: a context file with table facts computed by code, and one helper CLI exposing the tested calculators and a table-layout report as optional commands, with wording that they are never required and should be ignored if they do not fit.

Result on the same 21 items: 93.5 vs 93.0 bare (a tie), up from 84.1 orchestrated. 35 s and about 30 cents per item vs 31 s and 26 cents bare; the orchestrated version took 99 s. Every item the orchestrated build lost was recovered: the two wrong design labels, and the items where the fixed estimator returned nothing (the model noted the helper did not fit and used its own code). The model chose to run a helper on 9 of 21 items, mostly as a second number next to its own.

Takeaways:
- Optional tools cannot lower accuracy much; mandatory tools can. If you cannot show a gain from forcing the tool path, do not force it.
- A tie is the honest expectation where the bare model is near ceiling. Measure the wrapper on what it adds (missing-context cases, over-claim rate, audit trail), not on the benchmark the bare model already aces.
- Keep the arms like for like: same harness and prompt as the bare arm plus the additions, so any difference is attributable.
- Leak-audit gotcha: a substring filter on results/ flagged reads of the coding harness own tool-results folder (where it stores long command output). Match on full path segments, not substrings.
