Paired sign-flip test is conservative on 30-item benchmarks
With per-item differences of mean-over-runs (Q = 30 problems, 2-4 runs each), a Monte Carlo sign-flip permutation test with p = (1+count)/(1+B) rejected only 2-3% at nominal 5% when two identical systems were simulated (Rasch items, with or without item x system interaction). A paired z-test (mean diff / (SD/sqrt(Q))) rejected 5-6.5%. At Q = 111 both were near 5%. Many differences are exactly 0 and the rest take few distinct values, which makes the permutation distribution lumpy. When you compare test verdicts across error-bar choices, keep the test family fixed (z vs z), or the conservative permutation test will exaggerate the contrast. I nearly reported 65 vs 13 when the like-for-like figure was 65 vs 17.
Reproduced by simulation: two identical systems, Rasch items with difficulty SD 1.5 or 4 logits, ability 0.5, R=2 or 4 runs, interaction SD 0 or 0.5, statistic |sum d|, B=999, 4000 null datasets per cell (MC SE ~0.34 pp). Q=30: sign-flip with (1+count)/(1+B) rejected 2.0-3.6%, z-test 5.6-6.9%. So your numbers hold, but both tests are off, in opposite directions. Part of the z-test's excess is just the 1.96 critical value: a t-test with Q-1 df on the same differences rejected 4.0-5.3%. Most of the permutation test's shortfall is ties at the observed statistic: a mid-p version (count ties as half) rejected 3.8-4.9%. At Q=111 the plain sign-flip was still 3.1-3.8% with item SD 4 (4.6-4.8% with SD 1.5, R=4), so 'near 5%' depends on how many items are all-0 or all-1. For a like-for-like comparison at small Q I'd use t (df=Q-1) or mid-p sign-flip, not z.