qwen2.5:3b vs 1.5b for Ollama tool calling: 3b leaves optional args null and is 2.5x faster, but invents segment='all', city='California', limit=1 for 'highest'; its 60% was mostly substring hits
Same LangGraph + ChatOllama(temperature=0).bind_tools(...) agent and the same 10 questions over the HackPrinceton models_index.csv (14,944 rows), CPU-only Ollama, run back to back.
Headline numbers from the harness: - qwen2.5:1.5b (986 MB): 2/10, 542 s total (54 s per question; one answer that dumped JSON rows took 5.8 min). - qwen2.5:3b (1.9 GB): 6/10, 219 s total (22 s per question), one tool call per question, never zero.
3b is faster despite being twice the size because its answers are short. The 1.5b spent minutes generating copied JSON rows, so time per question tracks output length more than parameter count.
What 3b fixed: it left optional filters null (has_splat=None, has_drone_photos=None, min_pano_count=None), where 1.5b passed False/0 and silently over-filtered. So 'How many commercial models are in Houston?' -> search_index(segment='commercial', city='Houston') -> 6061 (correct).
What 3b still got wrong (all in the logged tool calls):
- Made-up enum values: search_index(segment='all', city='California', state='CA') for 'How many models in California have AI summaries?' -> 0 rows -> 'None of the models in California have been found'. Use Literal[...] types so the schema lists the allowed values, and treat 'all'/'any' as no filter in the tool.
- Missing fields: it had no ai_summary / defurnished filter to call. For 'models with defurnished views' it substituted has_splat=True and claimed '211 models'.
- 'Highest/most' questions: limit=1 with no sort -> 'highest panorama count ... is 61' (truth 422). The tool returned the first row, not the max. Add sort_by / top_by tools.
- Name by SID: called get_drone_photos(sid=...), then summarised the image list instead of the name. Add a get_model(sid) tool.
- Off-by-one semantics: 'more than 100 panoramas' -> min_pano_count=100 (>=) -> 234 vs 227. That passed only because of a 10% tolerance.
How much of '6/10' was real: Q4 (city with most residential models) passed because the answer described Houston listings, not because it compared cities. Q7 and Q10 passed because the model listed all 3-4 matching rows and the right name was somewhere in the text (substring grading). A strict grader gives about 3/10 for 3b vs 1/10 for 1.5b. Still a clear step up, but fix tool coverage and grading before comparing models further.