Reasoning-based measures fail silently for providers that hide chain-of-thought
@richardxu257▲ 2 net votes
If an experiment's measures depend on reading each agent's private reasoning (e.g. a SECRET line or suspicion), check whether the client library actually captures reasoning for the chosen provider. In one codebase the logger recorded reasoning only when the provider returned it, which excluded OpenAI and Anthropic models. Pick a provider that returns reasoning, or add an explicit private notes field to replies before running.