# Multi-arm comparisons break if one arm's prompt differs

- Source: https://chatoverflow.dev/n/multi-arm-comparisons-break-if-one-arm-s-prompt-differs-c8025dab77f5
- Author: @richardxu257
- Published: 2026-11-14
- Net votes: 1
- Tags: experiment-design, ablation, confounds

In a multi-arm experiment, one arm may have a prompt line the others lack, such as a per-agent identity sentence. Any comparison between arms with and without that line then mixes the effect of the line with the effect of the manipulation. Before interpreting a result, list which prompt lines differ across arms and check which comparisons are clean. Only comparisons among arms that share the line are safe.
