Optional helper tools that a strong model used on 22 of 24 tasks were used on 2 of 24 by a small model; hand small models the facts instead
Setup: an agent that audits causal claims on a table by writing and running its own Python, with an optional helper tool that reports how treatment is laid out in the table (threshold switches, within-unit changes, random-looking columns that move treatment). Same 24 simulated tables with known truth, plain arm vs arm with table facts in context plus the optional helper.
With a frontier model, the optional arm beat plain: it chose to call the layout helper on 22 of 24 tables and went from 1 over-claim and 2 wrong-direction answers to 0 and 0.
With gemini-3.5-flash-lite on the free tier, the same design did nothing: over-claims 6 of 12 plain vs 5 of 12 with helpers, missed real effects 3 vs 4 of 10. The model called the helper on only 2 of 24 tables; on one of those two it flipped a wrong supported (1.55) to inconclusive (0.25, truth 0). Its typical run was one regression script, then the answer, in 2 to 3 requests.
Lessons: - Optional-tool designs depend on the model deciding to look. Strong models do; small ones stop at the first plausible number. For a small model, compute the design-free facts up front and put them in the starting context rather than behind a tool. - Check tool-use counts before reading accuracy; a null result here was explained entirely by non-use. - Harness gotcha: a tool whose argument is a JSON string containing the answer JSON (double encoding) tripped the small model; it passed the object directly, got an error, and looped to the turn limit. Accept both shapes. - Free-tier practicalities that held: REST generateContent with responseMimeType application/json and a responseSchema as a text tool protocol; 164 requests at 10 per minute with no 429; about 20 s per task. Strip the API key read from the environment (a trailing newline raised an invalid-header error whose message contained the key) and scrub keys and tokens from the environment of any model-written code you execute.