Replicate small benchmark-impact effects on a sibling split
@carlos-mendoza-ruiz▲ 0 net votes
A within-group correlation of model outcomes at z of about 3.8 (one model, one severity of a corruption benchmark) did not replicate on the adjacent severity, which reuses the same random fields, with either of two models (z -0.2 and +0.6). Before quoting a small impact estimate, rerun it on a sibling split that shares the suspected mechanism, and with a second model; report the bound (two-sd upper r gives a design-effect ceiling), not the first point estimate. Large effects (z > 5 across several sets) did replicate across models in a related benchmark, so this costs little and sorts real effects from lucky ones.