Tool-orchestrated LLM scored below the bare model: the losses came from tool refusals and a guard that only accepts one tool output
Setup: a strong model either (a) writes its own analysis code, or (b) acts as orchestrator calling fixed statistical tools, with code-enforced guards on the final answer. Scored on 21 real benchmark items with reference designs and effects. Bare model 93.0; orchestrated 84.6; orchestrated plus extra tools (table-layout facts, cheap helper model writing one-off checks) 84.1, and 40 seconds and about 9 cents slower per item.
Where the gap came from, item by item: 1. Fixed estimator returned nothing on 3 of 21 items (data shape it does not support). The bare model just writes a different analysis. In one case the orchestrator DID get numbers from four helper-written checks, but a guard requires the cited interval to come from the fixed estimator tool, so the answer was forced to inconclusive with no estimate. A guard keyed to one tool name turns every gap in that tool into a zero. 2. Two items labelled observational where the reference says experiment or instrument, although the estimates were close. A prompt rule against assuming randomization or instrument validity unless told is the likely cause. 3. The added tools did nothing on these items because each comes with a prose study description that already states the design; they help only when that context is missing (confirmed separately on tables with anonymised headers).
Lessons: - Compare per item, not just composites; the same handful of items explained the whole gap for both orchestrated variants. - Every fixed tool needs a sanctioned fallback path that the guards also accept, or the system is strictly less capable than the model alone on anything off the happy path. - New tools should be evaluated on the inputs they were built for; on inputs where the needed context is already present they only add latency. - When re-scoring with a shared scorer that overwrites its report files, back up and restore them so a parallel session keeps its results.