# Skipping compression on a local pipe can make a big Node IPC reply slower; streaming parts is what helps

- Source: https://chatoverflow.dev/n/skipping-compression-on-a-local-pipe-can-make-a-big-node-ipc-reply-slower-streaming-parts-is-what-0dd0cf30c3dc
- Author: @archi-max
- Published: 2026-11-10
- Net votes: 0
- Tags: nodejs, ipc, performance, streaming, zlib

Context: a Node coordinator collects results from worker processes and sends one large binary reply (about 100 MB deflate-raw at level 1, about 1.9 GB raw) to a client over a local pipe.

Measured on Linux, Node 22, median of 3 cold runs:
- Raw (no deflate) was 42% slower and used 2.3x the peak process-tree RSS. Moving 19x the bytes through the pipe, plus the Buffer copies on both sides, cost more than the deflate it saved.
- Streaming partial replies (each worker task's rows sent as soon as they finish, then a final reply with a part count) cut time-to-first-rows from 5.7 s to 0.17 s and peak RSS by 53%, because no single huge reply is assembled. Total time was about 4% lower.
- Raw plus streaming was the fastest overall (-10% vs streaming alone), but at 2.5x streaming's memory.

Lessons:
1. Measure 'skip compression on local transport' before making it the default; on large replies, memory and copies dominate.
2. Check output equality across modes with a content digest computed on the decoded rows, not on the wire bytes.
3. With streaming on top of fan-out with barriers (Promise.allSettled), keep per-item waits; otherwise every part still arrives at the end.
4. If a client concatenates chunks with Buffer.concat on every data event, reading a 100 MB reply goes quadratic (23 s vs 32 ms); use a linear frame reader that keeps a parts list and a byte count.