New code path looked 165x slower than the old one: it was run with a shorter options list, not a design flaw
Situation: a GPU engine was moved from a stand-alone program into a library loaded by another program. On small inputs the library gave identical results. On the first large input the job ran 13+ minutes at 100% GPU utilisation and hit its time limit; the stand-alone program does the same input in under 2 minutes. It looked like the new in-process design had a per-call cost problem.
What it was: the test kit passed the library two option flags, while the stand-alone program is always run with about twenty, several of which switch on the fast output path (batched emit, async and pooled writers, worker threads). Without them the engine silently falls back to an old slow emit path. Nobody chose that; the kit author simply wrote the minimal list that made it run.
How it was found without another long run: - The engine prints per-stage times. On a small input with the short list, the emit stage was 9.8 s of a 14.3 s run for 42 million records (4.3 M/s); the compute kernels were 0.07 s. An earlier record of the stand-alone program on the big input showed 720 M records/s. - Arithmetic: the big input has 3.96 billion records; at 4.3 M/s that is ~15 minutes for emit alone, which matched the hang. - Confirmed by one short run of the library with the full list on the same small input: emit 0.056 s for the same 42 million records.
What I changed: - One authoritative options file, one line per option with value, which variant it applies to, and where it comes from; every kit reads it, and each result line prints its hash. A script checks it against the options of the last accepted run. - A comparison is never run until both arms are shown to use that file; the party who owns the engine signs off the list the other party proposes. - Jobs print a progress line per unit of work, so a slow run shows a rate instead of silence. - Fast paths that are opt-in by flag are a trap: at minimum the engine should print which emit path it took in its first lines.