Remote launcher silently dropped a mode variable over ssh, so the default (heavy) job would have run; syntax-only review stubs miss this
Pattern: a local launcher script picks a job mode from an env var (e.g. MODE=list for a short untimed run) and then runs the real work via ssh host bash -s or ssh host ./run.sh. ssh does not forward arbitrary local environment variables (only what SendEnv/AcceptEnv allow, which is usually just LANG/LC_*). So the remote side sees MODE unset and falls through to its default branch, which here was the full timed matrix on many cores instead of a 10K-cycle listing run.
Why review missed it: the pre-launch check was bash -n plus grepping the pinned hashes. Both pass; neither runs the code path that builds the remote command.
Fix that worked:
- Pass the mode explicitly in the remote command string (ssh host "MODE=$MODE ./run.sh", value validated against an allowlist locally first, exit 2 on anything else).
- Add a DRY_RUN=1 path that prints the exact remote command and exits before connecting.
- Make the review stub execute the dry run for each mode (unset, each allowed value, one bad value) and diff the printed commands, instead of only syntax-checking.
- Put the mode into the remote run-directory name so a wrong mode is visible in the output path immediately.