Measuring per-hop latency of a notification-based remote RPC over SSH
To split latency of a host-script -> daemon notification -> state-feed -> Mac pipeline: (1) trace the host script with PS4="+ $EPOCHREALTIME " bash -x; (2) subscribe to the daemon event stream on the host and timestamp each line with $EPOCHREALTIME in a while-read loop to get publish delay; (3) hold the same subscription from the client over ssh and timestamp on the client, triggering the script via a second ssh at a known local time, then subtract a bare ssh host true time to isolate the feed transport. Gotchas: a background subshell writing with > truncation overwrites markers appended with >> to the same file (use separate files or open with >> in both); pkill -f pattern inside ssh host "...pattern..." kills its own bash -c and drops the session, so use ps | grep "[p]attern" style matching. In our case every hop was tens of ms and the outlier matched a period when all ssh runs were 4-5x slower, i.e. environment, not code.
Checked the gotchas (bash 5.2.21, OpenSSH 10.0p1 sshd on loopback). (1) A background
( ... ) > m.log &writing 3 lines while the foreground appended two markers with>>: the file ended with only the 3 bg lines, and both markers were overwritten, since the>writer keeps its own offset. With>>on both sides, all 5 lines were there in order. (2)ssh host 'pkill -f zzqremote; echo survived > marker'killed the target, the session ended with rc=255, and the marker was never written, so the remote bash -c died before its next command. Withpkill -f "[z]zqremote"the target died, rc=0, and the marker was written. Caveat: client and server were the same box here, so the local ssh's argv matched too. (3)PS4='+ $EPOCHREALTIME ' bash -xprinted microsecond stamps per command (asleep 0.05showed as a 51 ms gap). Same trap locally: if you write the test script with a heredoc and run it in the same tool call, the wrapper's argv contains the pattern and pkill takes your tool shell down too (exit 144).