Skipping compression on a local pipe can make a big Node IPC reply slower; streaming parts is what helps

Context: a Node coordinator collects results from worker processes and sends one large binary reply (about 100 MB deflate-raw at level 1, about 1.9 GB raw) to a client over a local pipe.

Measured on Linux, Node 22, median of 3 cold runs: - Raw (no deflate) was 42% slower and used 2.3x the peak process-tree RSS. Moving 19x the bytes through the pipe, plus the Buffer copies on both sides, cost more than the deflate it saved. - Streaming partial replies (each worker task's rows sent as soon as they finish, then a final reply with a part count) cut time-to-first-rows from 5.7 s to 0.17 s and peak RSS by 53%, because no single huge reply is assembled. Total time was about 4% lower. - Raw plus streaming was the fastest overall (-10% vs streaming alone), but at 2.5x streaming's memory.

Lessons: 1. Measure 'skip compression on local transport' before making it the default; on large replies, memory and copies dominate. 2. Check output equality across modes with a content digest computed on the decoded rows, not on the wire bytes. 3. With streaming on top of fan-out with barriers (Promise.allSettled), keep per-item waits; otherwise every part still arrives at the end. 4. If a client concatenates chunks with Buffer.concat on every data event, reading a 100 MB reply goes quadratic (23 s vs 32 ms); use a linear frame reader that keeps a parts list and a byte count.