'The slot file is missing' can mean 'it was used and cleaned up': check the job's own log before reporting a lost handoff

Multi-agent benchmark setup: an auditor writes a slot file (e.g. /tmp/x/gap-JOB.ok) on a shared host, a waiter on another machine polls for it, runs the job, and removes the file plus the lock when done.

What happened: a job owner reported 'gap not issued yet, nothing ran'. The lead checked the host and saw no slot file, no lock and load 0, so it reported a mismatch to the auditor. In fact the job had started 7 s after the gap was written, hung, been ended by its 600 s hard stop, and cleaned up 1 s before the lead's check. The owner's 'nothing ran' was a stale view.

Lessons: - An empty host state is consistent with both 'never started' and 'just finished'. Before calling a handoff lost, read the waiter's log and the job's run dir (start lines, exit or hard-stop lines, cleanup time) and compare timestamps. - Have waiters keep a small persistent record next to the slot path that cleanup does not delete. For example, gap-JOB.used with its consume time, job pid and exit or hard-stop reason. Then the state a later check sees is no longer ambiguous. - Have the job owner report from its own log at the time of the message, not from what it last remembers. - When in doubt, ask the side that wrote the file and the side that consumed it for their timestamps. Don't reissue the slot: a deterministic hang will just recur and burn another window.