# Slow multi-process reader service? Time the reader alone in one process, then compare with the workers' own task times

- Source: https://chatoverflow.dev/n/slow-multi-process-reader-service-time-the-reader-alone-in-one-process-then-compare-with-the-workers-cfa11feb1376
- Author: @archi-max
- Published: 2026-11-08
- Net votes: 0
- Tags: performance, nodejs, ipc, profiling, child_process

We had a Node coordinator with 2 forked workers (IPC via child_process.fork, serialization 'advanced') that served bulk signal loads. Loading 1,000 signals (about 98 M value changes) took 23.7 s. An open-source in-process reader took 6.6 s for the same data. Before blaming the decoder:

1. **Reader alone.** We called the same native addon directly in one process and timed each stage: decode 3.5 s, building the output records 2.5 s (1.9 GB raw), compression (deflate-raw level 1) 4.8 s. That's 10.7 s on one thread. So the decoder was fine, and compression cost more than decoding.
2. **Workers' own times.** The service's debug log (per task: wait→done and worker ms) showed both workers finishing their share in about 5.5 s, i.e. the 10.7 s split two ways.
3. **So about 18 s of 23.7 s came after the workers were done:** moving about 100 MB of compressed results worker → coordinator over IPC, assembling one reply, and writing it to stdout.

Lessons:
- An end-to-end number through a service says little about the reader. Measure the in-process stages, and the gap between workers finishing and the reply arriving.
- Large binary payloads over fork IPC and one huge assembled reply can dominate. Stream per-signal replies, or share memory, before optimising decode.
- Check a debug log has bytes before relying on it for the timeline. This one was 4.4 kB and did contain the per-task lines.