Misspecified effect prior manufactures subgroup residuals in selection models
When asking whether a subgroup (e.g. one field) replicates worse than a publication-filter model predicts, the answer can hinge on the prior family for true effects. In a replication dataset, a normal prior fitted per subgroup flagged a 'significant' residual subgroup gap (p = .006), but a spike+normal or nonparametric (NPMLE) prior, which fit that subgroup's originals much better (LR ~12), left none. Decisive check: simulate a world where the filter is the ONLY mechanism, with the subgroup's effects drawn from the better-fitting spike fit, then refit with the normal prior. It reported a residual of about -0.24 on average where the truth was 0. Before claiming a moderator from a residual, fit at least one flexible effect distribution and run that known-truth sim. A pile-up of z/SE just above 1.96 reads as 'many near-null tested effects' and is also the p-hacking signature, so the model cannot separate the two.
The mechanism reproduces in a generic simulation; your data not re-checked. Setup on the z scale (se = 1): 40 published originals per subgroup, filtered at |z| > 1.96, replication z_R ~ N(theta, 1). Residual = mean sign(z) * (z_R - predicted), with the prediction from the posterior mean under a prior fitted to the originals by truncated MLE. 300 reps each, mean residual (sd across reps):
truth spike pi0=0.6 + N(0, 2.5): normal prior -0.16 (0.25), spike+normal prior +0.04 (0.29) truth spike pi0=0.8 + N(0, 2): normal prior -0.40 (0.29), spike+normal prior -0.03 (0.34) truth plain N(0, 1.5): normal prior +0.03 (0.29), spike+normal prior +0.11 (0.33)
So a normal prior does manufacture a negative residual when the truth has a null spike, and it grows with pi0. The flexible prior is not free, though. When the truth is plain normal, the spike+normal fit on 40 studies over-shrinks and overpredicts the decline by 0.11 (SE about 0.02 over 300 reps). Your known-truth check should therefore be run in both directions, simulating from each candidate prior and refitting with the other.