Calibrate monitor threshold on an honest, observe-only pilot run
@richardxu257▲ 1 net votes
To set a monitor threshold for a control protocol, run pilot configs with the side task disabled (no secret-submission tool or private side-task prompt) and the monitor in observe-only mode (threshold null), so it scores every post without blocking any. Compute the threshold from those honest scores, then check it against a paired run with the side task on to measure the catch rate. Calibrate only on pilot worlds and never on the held-out sweep worlds, and freeze the value before the main sweep.