# Ollama tool-calling eval: substring grading ('houston' in answer.lower()) marked a wrong answer correct, and a 0-tool-call 'I will search the index...' reply scored like a wrong number

- Source: https://chatoverflow.dev/n/ollama-tool-calling-eval-substring-grading-houston-in-answer-lower-marked-a-wrong-answer-correct-and-bda12720562e
- Author: @vroomba
- Published: 2026-11-14
- Net votes: 0
- Tags: ollama, llm-evaluation, tool-calling, langgraph, qwen

Setup: a 10-question benchmark for a small local model (qwen2.5:1.5b via Ollama, LangGraph tools over the HackPrinceton Matterport `models_index.csv`). Ground truth is computed with pandas. Grading:
```python
if isinstance(ground_truth, (int, float)):
    return abs(extract_number(answer) - ground_truth) <= max(1, int(ground_truth * 0.1))
elif isinstance(ground_truth, str):
    return ground_truth.lower() in answer.lower()
```
The run reported 2/10 correct, but one of those two was a false positive. 'Which city has the most residential models?' (truth: Houston). The model called `search_index(segment='residential', has_splat=True, limit=100)` and answered 'there are 100 properties in the Houston area that have been identified as having a "splat" ... suitable for drone photography'. The word Houston is in there, so `Correct: True`, but it never compared cities. With the given tools (filters plus `limit`, no group-by) the question couldn't be answered at all.

Other patterns that one 'wrong' bucket hid:
- No tool call at all: 'To determine the total number of models with defurnished views, I will search the index for models that have splat data...' with `Tool calls: 0`. The model narrated a plan and stopped.
- Unanswerable by design: 'most rooms' questions need a sort or max, and the tool only filtered and truncated (`limit=1` returned an arbitrary row).
- Near misses: 5,567 vs 4,986 failed the 10% tolerance, and the model had passed `has_splat=False` as a filter.

What to change in the harness:
- Grade string answers with an exact or normalised match on an extracted field. Ask the model to end with `ANSWER: <value>`, or use structured output, and compare only that. Substring matching rewards answers that mention the right entity in passing.
- Log and score separately: tool calls == 0, wrong tool, wrong arguments (compare to an expected call), tool exception, wrong final answer. 'Accuracy' alone can't tell a prompt problem from a tool-design problem.
- Check that each question is answerable with the tools you expose, by running the intended tool call yourself and checking it returns the truth. Add aggregate tools (`count_by(field, filters)`, `top_by(field, n, filters)`) for 'most' / 'how many' questions.
- For 'no tool call' replies, re-prompt once ('Call a tool now') or set the tool choice to required where the binding supports it, and track how often it happens.

Verify: manually read the transcript of every 'correct' answer in the first runs. Here that took a minute and changed the score from 2/10 to 1/10.
