Finding a 2x decode regression between two versions of a header-only reader: staged copies with a mismatch check

Situation: version B of a header-only decoder was ~2x slower than version A on point queries, even on byte-identical compressed blocks. Sampling profilers on macOS just showed one big inlined function.

What worked: 1. Make both versions decode the same input, so layout or encoder differences are ruled out. Write the same records with both and check the block sizes and row counts match. 2. Copy each version's hot function into a profiling harness, split into stages (decompress, value decode, delta walk, bookkeeping, validation). Time each stage with steady_clock over many blocks. 3. After each staged run, compare the staged output with the library's own output (mismatches must be 0). That way you know the split didn't change the work. 4. Then try the obvious variants of the worst stage, with the same checks kept: pre-sized arrays instead of push_back, batched varint decoding (read up to 10 bytes at once) instead of a per-byte checked getter.

What it found: ~25% was a per-row push_back into an offsets vector that the caller never used for fixed-width values. The delta walk was 2.7x slower than A's, because of a per-byte bounds-checked read plus push_back. The batched walk with identical checks matched A. The new validation that was the suspect (a mask check) was only 3.5%.

Lesson: when a correctness change lands with a slowdown, measure it in stages before blaming the checks. The cost is often incidental bookkeeping added at the same time.