`ssh host cat file > local` can stay alive after all bytes arrive; verify by size+hash instead of waiting for the script's last log line
Pattern: a copy script streams a large file with ssh host "cat path" > out &, waits on the ssh pid, then logs size + sha256 and prints COPIED. Other watchers wait on that script's pid or its final log line.
Observed (macOS client to Linux host, 2.8 GB): the local file reached the exact expected size and its sha256 matched, but the ssh process was still alive minutes later. So the script never printed COPIED, and two watcher loops kept waiting on it. Lingering ssh sessions like this are commonly caused by a ControlMaster/multiplexed connection, a remote child holding the channel open, or a slow channel close under a busy remote host.
Practices:
- Before asking for an approval or waiting on a 'stuck' copy, check whether the work has already happened. Compare the output file's byte count and hash against the known values. An approval request for something already done just costs a round-trip.
- In the script, don't rely only on the ssh pid exiting. Also poll the output size, and when it reaches the expected byte count, hash it and finish. Or add -o ServerAliveInterval=15 -o ServerAliveCountMax=3, a timeout on the transfer, or rsync/scp, which close cleanly and can resume.
- Use ssh -T (no tty) and avoid multiplexing (-o ControlMaster=no -o ControlPath=none) for bulk transfers when other sessions share a control socket.
- Kill only the recorded pids of your own script, never a pattern match.