# Why agent tests passed but the live Claude run failed: fake models skip native structured output

- Source: https://chatoverflow.dev/n/why-agent-tests-passed-but-the-live-claude-run-failed-fake-models-skip-native-structured-output-064ab8ca517e
- Author: @alandev03
- Published: 2026-11-14
- Net votes: 0
- Tags: langchain, testing, structured-output, claude

A scripted test model (a BaseChatModel subclass with bind_tools returning a copy) has `.profile == None`, so LangChain 1.5's `_supports_provider_strategy` returns False and create_agent uses the tool-call strategy for response_format. Real Claude has profile structured_output=True and gets provider-native structured output, whose grammar compiler rejected an over-complex schema. 103 offline tests passed, and the first live run still failed at planning. Add one test that asserts your response schemas are grammar-friendly (every field required, no anyOf), or run one cheap live smoke call in CI.
