Per-call capture hooks with a consumer thread: profile for false sharing before tuning the copy path
Setup: a simulator calls a C capture hook many times per cycle. The hook copies values into a buffer, and a separate consumer thread encodes and writes them. The hook's cost on the simulation thread is what matters.
We broke the per-call cost into parts with a fake host (a stub that implements the simulator's callback API) linked against the real hook library, using arms like these: - empty hook (the floor); - the real hook with a consumer that discards records (the producer's own intrinsic cost); - the real hook with the real consumer; - the real hook with a layout-only patch. Sweep the payload size to get the per-byte cost.
Result (diagnostic, Apple silicon container, ~114 B/call): - Per-byte copy cost: about 2-3 ns/call. Negligible. - Intrinsic producer path: 12-13 ns/call. - False sharing with the consumer thread: 18-35 ns/call, the largest single piece. The hot path read three cache lines per call that the consumer wrote on every record: a callback counter sharing a line with the consumer's event counters, a vector header next to a dirty list, and per-group data next to per-group dirty flags. - Other interference from the consumer running at the same time: about 20 ns/call. Platform-dependent, so measure it on the target host.
Fix: a layout-only change. Separate producer-read fields from consumer-written fields into their own 64-byte-aligned lines (alignas(64), or padding). Verify the output stays byte-identical. It won 14/15 paired reps.
Takeaways: - Before optimising the copy or encoding, check which lines the hot path touches that the other thread writes (code inspection plus perf c2c on Linux). - Cross-core line transfers cost several times more on multi-socket or server parts than on a laptop, so confirm on the real host.
Measurement note for anyone repeating this in a Linux container on Apple silicon. Per-run CPU per call swung 2-3x (e.g. 62-160 ns) between runs of the same binary, probably because the VM's vCPUs land on performance or efficiency cores and the producer stalls on queue back-pressure. Medians from separate sessions also moved: the same build gave 35 ns in one session and 52 ns in another. What made the A/B readable: (1) build every variant first, then run them interleaved rep by rep, and compare paired per-rep differences, not medians from different sessions; (2) a control where the consumer discards records, which gives tight spreads and shows whether a change touches the producer path itself or only cross-thread effects; (3) no hardware counters in the VM, but 'perf record -e cpu-clock' does work after installing linux-perf in the container (it needs --cap-add SYS_ADMIN and seccomp=unconfined). Splitting samples by thread showed the host's own code on the producer thread running about 2x slower while the consumer worked.