Publishing a benchmark harness that also benchmarks a proprietary tool: generic extension hooks + clean-checkout test
Goal: open-source a file-format benchmark whose private version also measures a closed-source reader (vendor libraries, paths and API you can't publish).
What worked:
- Replace the proprietary arm with generic hooks the public tree owns:
- an optional compile-time include (-DQBOP_EXT_READER=path.h) implementing a small Reader interface;
- build-script env vars (QBOP_EXT_*) that, when set, build an extra binary;
- a JSON spec file (QB_EXT_ARM=spec.json) giving converter commands and flags, read at run time;
- neutral result keys (ext_load rather than the tool's name).
- Keep the real include, build wrapper and spec in a private directory that the public tree never references, not even in .gitignore comments.
- Test from a clean export (git checkout-index -a --prefix=/tmp/x/): the build must succeed with the hook absent, and a dummy include must build and run through the hook.
- Before committing, grep the branch and the commit message for host names, user names, private IPs, internal paths, vendor and tool names, API symbols and licence terms. Expect a few false positives, e.g. /sys/class/dmi and -ffreestanding for a 'class' or 'free' pattern.
- Drop vendored copies of internal docs that the build doesn't need.
Unrelated gotcha found while writing the results: report contention inflation as concurrent/serial − 1 (1/ratio − 1), not 1 − ratio. A serial/concurrent ratio of 0.63 means +59% inflation, not +37%.