Verify a memory-budget scheduler with an external sampler: its own 'peak' log can be taken after it reacts
Reviewing a job scheduler that keeps the summed RSS of its worker processes under a budget (stop the youngest worker when over, requeue it). The scheduler's own end-of-run line said 'peak measured 74 MB' against a 60 MB budget. An external sampler running alongside (ps over the process tree every 50 ms) showed 96-101 MB. The scheduler recorded its peak after it had stopped a worker, so the logged value was the post-reaction total. Its own stop line printed 98.
Other things the external sampler showed that the scheduler's logs hid: - 'Over budget for at most one poll' was really two polls, because each poll stopped only one worker while several had started together, unmeasured. - An inherited test-only environment variable that fakes RSS readings silently disabled the whole backstop (the logs said 0 MB).
Practice: - Assert the bound in tests using a sampler outside the scheduler, at a faster rate than its poll. - Record the true maximum before reacting, and log it. - Test with the real worker binary, not a stand-in that ignores its cap. - Refuse test-only hooks unless an explicit test flag is passed.
Cost of the fix: gating every start on the running workers having a few measured polls makes starts serial. With ~1 s per start, the estimate for a big many-shard job was +20% wall time at a 1 s poll and ~+7% at 0.25 s. Mitigation: skip the warmup gate while the admitted total plus 2x the new worker's prediction is well under the budget (we used 50%), so only tight-margin starts are serialised. Use a short poll interval on big hosts.
Related case: a fault-injection runtime switch (a +plusarg that corrupts a self-check so the 'untrusted' path can be tested) was accepted by every normal build. A reviewer called it harmless because it can only make the check fail. Still, one stray argument in a real regression run would mark good results untrusted. The fix was to compile the switch in only with a test-only define, and have every other build refuse it with a clear error and non-zero exit. Add a red test that passes the switch to a normal build and expects the refusal.