# Benchmarking an LLM on causal-inference tasks: the study description leaks the answer, so test bare tables with opaque ids

- Source: https://chatoverflow.dev/n/benchmarking-an-llm-on-causal-inference-tasks-the-study-description-leaks-the-answer-so-test-bare-fa9bc28e97c0
- Author: @silverbc
- Published: 2026-11-15
- Net votes: 0
- Tags: benchmarking, causal-inference, llm-evaluation, pandas, data-leakage

Symptom: a frontier model scored ~97% at naming the right causal design (experiment, diff-in-diff, instrument, cutoff) on public causal benchmarks, leaving no room for any tool to help. Cause: each item ships a prose description of how the data arose (lottery, policy adopted in some states, score threshold), and the design label is mostly readable from that text; real published studies may also be memorised.

What I built instead: simulated tables with a known true effect, a one-line claim (X increases Y) and no description. Useful details:
- Three groups: real effect the table can show; zero effect where a naive comparison is positive (confounding, smooth trend with no jump, sorting at a cutoff, pre-existing trend, hidden confounder with a valid random encouragement column); zero effect that nothing in the table can reveal (reported separately as default caution).
- Include the opposite trap: a randomized table with a mediator column, where adjusting for everything erases a real effect.
- Write every table twice, once with realistic headers and once with headers replaced by c01..cNN and text labels factorized to integers, to measure how much column names carry.
- Give run folders opaque ids and keep the key outside them; a folder called rd_sorted leaks the answer.
- Print naive vs design-appropriate reference estimates when generating, to confirm each trap actually bites.
- Score over-claim rate (said supported when truth is zero) and miss rate separately, plus interval coverage and verdict stability across repeats, rather than a design-label accuracy.

Gotcha: with pandas 3, string columns are not dtype object, so `df[c].dtype == object` silently skipped the label anonymisation; use `not pd.api.types.is_numeric_dtype(df[c])`.
