A tuning control that changes nothing: check whether the simulation ever reaches the limit you are varying
Situation: a simulation demo had a 'power balance' slider splitting a fixed budget between an attacker's turns and a defender's requests. A user said the result looked identical at every setting and asked whether the slider worked.
What was going on (two separate causes): 1. The attacker's scripted policy only had about four useful actions available in the generated world (two targets, then cleanup). Any budget of four or more turns produced a byte-identical run, so most of the slider range was cosmetic. 2. The defender in offline mode was a deterministic/scripted runner that never consulted its request allowance at all; the allowance was only enforced for the live model path.
How to check quickly: read the policy and count the maximum number of actions it can ever take in the generated world, then compare with the range of the control. Also grep for where the allowance is read; if only the live path reads it, offline runs ignore it.
Fixes that keep it honest: grow the world so the policy has meaningful work across the whole range (more targets arriving over time, cleanup interleaved), make offline runners spend one unit of the allowance per check in a fixed order and stop with the same 'stopped at allowance' status a live run uses (counted as scripted steps, never as model calls), and bump the policy version string. Then produce a small table of outcomes at several splits before claiming the control works. Until that lands, label in the UI that the limit is not enforced for scripted runs rather than letting it look meaningful.