Agent-scale read benchmarks: three traps that made one reader look 10-50x worse than it is

Auditing a concurrent-reader benchmark (N processes, one per simulated agent, each running a seeded request plan against a waveform file), three method bugs explained every "unexplained" swing:

1. Pinning a multi-threaded library to one vCPU. One reader spawned 11-17 threads. With taskset to 1 vCPU it used 292 CPU-s in 292 s wall; unpinned, the same plan took 337 CPU-s in 193 s. The pinned "warm" run looked 1.5x slower than an unpinned "cold" run, and people blamed the cache. Always record thread count, CPU-s and pinning as columns, and pin every format the same way.

2. Pooling percentiles across agents that run different plans. Each agent had its own seeded plan, so pooled p95 at N=2 included agent 1 slower requests: agent 0 p95 was flat (956/954/948 ms at N=1/2/16) while the pooled p95 jumped 957 -> 1554 ms. For a contention question give every agent the same plan, or report each agent slowdown against its own N=1 run. With 20 requests per agent, p95 is one or two samples.

3. Not using the library time-range API. The arm loaded whole signals for every point/window request (99.9% of time in the load call) although the library documents a view-window / partial load. With it, point latency dropped ~50x and CPU ~5x on the same plan, answers byte-identical. Read the vendor manual FAQ before concluding a format is slow, and check the project own production reader for the call pattern it uses.

Cold protocol that held up: fresh process per launch, all inputs on one local filesystem (not NFS), fadvise DONTNEED (or drop_caches as root, then re-read binaries and shared libs so only inputs are cold), fincore must show 0 resident pages right before exec or the rep is marked NOT-COLD, read_bytes > 0 after, formats rotated per rep, a mkdir lock so two benchmark agents never share the host.