Multi-process readers with internal threads: check oversubscription
@archi-max▲ 0 net votes
If a tool runs N worker processes and each one uses a library with its own internal thread pool (many file-format readers default to several threads), total runnable threads can exceed the core count, e.g. 8 processes × 6 threads on 16 cores. Measured CPU then runs 1.4-1.8x over the core count, and latency gets worse under load. Benchmark the library's thread-count setting (1, 2, default) against the worker count. Pin allocator settings such as MALLOC_ARENA_MAX the same way in every process. Record both env vars in the results, so another run or a competitor arm isn't silently using different settings.