Moving a per-event writer off a hot producer thread: SPSC chunk ring + replay gives byte-identical output; measure thread CPU and process CPU separately

Context: a callback-driven producer (a simulator thread) called a pipelined file writer's per-event change() directly; the writer's per-event work (per-signal row vectors, so a random cache line per event, plus occasional realloc, job allocation and mutex enqueue) ran on the hot thread.

What worked: 1. Put a single-producer/single-consumer ring of fixed-size chunks (e.g. 16 x 1 MiB, allocated and budget-reserved once) in front of the writer. The producer appends [id|len][value padded to 8] records and writes a time record only when the time changes. No allocation, lock or clock query on the hot path. A chunk is published when the next record doesn't fit, or at a control op (cut/close). 2. One dispatcher thread replays exactly the same change/cut/close calls on the existing writer. The output is then byte-identical to the direct path by construction. Check it anyway: sha of the files from the direct, ring, and ring-with-deferred-setup paths, in unit tests and every benchmark run. 3. Two Lamport index queues (full/free) with capacity >= pool size. Spin briefly, then park on a condvar; wake only if the other side announced it is parked (seq_cst flag + fence). Count producer waits: that is the only way the writer can slow the producer. 4. Values larger than a chunk: reserve in the memory budget first, heap-copy, and pass a pointer record. Release on the consumer. 5. Run setup (dictionary/metadata commit, worker start) on the dispatcher, so the producer starts immediately and records buffer in the ring.

Measured on a synthetic 78K-signal / 9M-event run (Apple M4 Pro, loaded machine, n=3): producer-thread cost per event dropped from ~48 ns to ~6-11 ns, but process CPU rose 12-18%. Report both, since moving work off-thread is not free. Use CLOCK_THREAD_CPUTIME_ID for the producer and getrusage for the process.

Gotchas: (a) with no other work on the producer, the dispatcher becomes the bottleneck and the producer waits on a full pool; (b) whatever the writer must do at close (here, encoding one tiny final block per signal) still lands on the producer's wall, so time the close phase separately; (c) macOS has no fdatasync declaration (compile with -Dfdatasync=fsync for local tests); (d) test mutants: drop the time record, skip publishing the last chunk, leak spill budget, reset the time per chunk. A good order/budget test catches all four.