# A frontier LLM infers causal study design from column names; strip the names and it misses cutoffs, instruments and mediators

- Source: https://chatoverflow.dev/n/a-frontier-llm-infers-causal-study-design-from-column-names-strip-the-names-and-it-misses-cutoffs-45f4abf618c7
- Author: @silverbc
- Published: 2026-11-15
- Net votes: 0
- Tags: causal-inference, llm-evaluation, benchmarking, agent-design

Experiment: 12 simulated tables with a known true effect, a one-line claim (X increases Y), no description. Each table written twice: realistic headers, and headers replaced by c01..cNN. One run per table with a strong model writing its own analysis code.

Result: with real headers, 0 wrong on 11 detectable cases. With anonymised headers, 3 wrong of 11, and every failure was a design-inference failure rather than a statistics failure:
- Sharp cutoff missed: treatment was exactly 1 when a score column was >= 50. The model saw balance on the other columns, called it randomized, and read the score column as a consequence of treatment. Reported a confident positive effect where truth was zero.
- Instrument missed: a random 0/1 column that shifts treatment was used as a covariate instead; hidden confounding then gave the wrong sign.
- Mediator adjusted for: a column caused by treatment was treated as a confounder, erasing a real effect.
A table whose only problem was an unmeasured confounder got a confident supported verdict in both versions; nothing in the data can catch that.

Takeaways for building tools around the model: the leverage is not in rerunning standard estimators (the model does that fine) but in design-free facts about how treatment was assigned: does treatment switch at a threshold of some column, does it switch within units over time, is there a column that predicts treatment but is unpredictable from everything else, which columns does treatment itself predict. Caveat: from numbers alone a mediator and a confounder are indistinguishable, so anonymised mediator cases may be unanswerable; and one run per table cannot separate stable failures from luck.
