Sharding one file writer across N encoder threads: per-key order, lock the commit, make global events barriers, compare decoded streams

Pattern that worked for a block-structured file writer (one output file, many independent streams keyed by ID) to spread encoding over N threads with no separate committer thread: - Partition by stream ID (id % N or by activity). Each shard is a full single-threaded pipeline (buffer, encode, commit) on its own thread, so no queue or hand-off. - Commits go into the one writer under a mutex. This is valid only if the writer requires ordering per stream (and against global cut points), not across streams. Check the writer's append preconditions first. - Global events (segment cut, pause marker, close) become a barrier. Every shard gets the event at the same point in its own input. The last shard to arrive runs the writer call under the lock, and the others wait on a generation counter. - Failure handling: any shard's failure or abandon sets a group abort flag and notifies the barrier condition variable, so waiters return false instead of hanging. Commits check the flag before touching the writer. Test this with a watchdog thread that _exits on a hang. - Equality testing: with N>1, the interleaving of blocks across streams follows thread timing, so file bytes vary from run to run. Compare per-stream decoded digests and per-stream block layout against the single-thread run, plus the reader's full verify. Keep N=1 byte-identical as a regression anchor. - Memory: each shard keeps its own per-stream table and encoder workspace, so budget N copies.