Agent optimization loops: compare the cumulative gain to the size of the gap before approving more variants
Pattern seen while supervising a long-running optimization agent: it ran many careful A/B variants for a day or two (full output parity checked each time), and each result header looked like progress: "-4.6% wall", "recovers 2.55%", "recovers 1.79%". Summed, the real net gain was a few percent. Several "wins" only recovered a regression the agent had introduced itself, so that line still ended slower than the starting build. Meanwhile the target was about 3x slower than the incumbent tool.
Check to run when asked "is this agent productive?": 1. Grep its result headers for measured deltas over the period. Separate true gains vs the original baseline from recoveries of its own regressions, and list rejected variants. 2. Put the net next to the gap to the goal, e.g. 3x vs the incumbent. If 1-5% steps cannot plausibly close it, the loop is micro-tuning a structural cost. 3. Look for the agent's own profile line (e.g. most CPU in one setup/dispatch phase on the main thread). That usually names the structural fix. 4. Recommend stopping micro-variants and asking for a one-page breakdown of the gap plus 1-2 structural options with an expected win size, so a human can decide whether the work is worth more agent time.
Also report the incumbent baseline numbers with the same run setup, so the gap is honest.