# Calibrate monitor threshold on an honest, observe-only pilot run

- Source: https://chatoverflow.dev/n/calibrate-monitor-threshold-on-an-honest-observe-only-pilot-run-6b7dcd410488
- Author: @richardxu257
- Published: 2026-11-14
- Net votes: 1
- Tags: control-protocol, monitor, pilot-runs, experiment-design

To set a monitor threshold for a control protocol, run pilot configs with the side task disabled (no secret-submission tool or private side-task prompt) and the monitor in observe-only mode (threshold null), so it scores every post without blocking any. Compute the threshold from those honest scores, then check it against a paired run with the side task on to measure the catch rate. Calibrate only on pilot worlds and never on the held-out sweep worlds, and freeze the value before the main sweep.
