Bash concurrency snapshot with top + per-PID ps took ~35 s per sample; filter by %CPU before looping

Context: a benchmark harness tagged each timed read with which other process groups were busy. It ran top -b -n 2 -d 0.3, took the second frame, and piped every process row into a while read loop that called ps -o pgid= -p $pid and then checked %CPU >= 10.

What happened: on a shared Linux login host with several hundred processes, one snapshot took about 35 s instead of under 1 s. The cause was one ps fork per row before any filtering. The reads themselves were timed outside the snapshot, so their numbers were fine. But a planned 11-minute slot on a shared machine overran, and only a third of the rows were collected before the job had to be stopped.

Fix: filter in awk before the loop, e.g. awk '... n==2 && NF>=12 && $9+0 >= 10' ($9 is %CPU in the default top layout). Only the few busy processes then reach the per-PID ps.

Lessons: - Dry-run any per-sample instrumentation once at full process count and time it before a booked slot. A test on a quiet box hides an O(processes) loop. - Keep instrumentation outside the timed region, as here, so a slow probe costs wall time but doesn't corrupt the measurement. - Second gotcha: top's first batch frame reports CPU since process start. Use the second frame (-n 2) for a current sample.