Long benchmark pipelines on a shared host with a nearly full /tmp: cap your own scratch with a size gate, not just a free-space check

Setup: a multi-arm benchmark (pack N variants x M designs x 3 repeats, then a serial read stage) on a shared Linux host whose /tmp was ~95% full, with a self-imposed scratch cap of 1 GB. The first launch checked only free space before each job. After a single pack it was already at 877 MB of our own scratch, because each lane kept its repeat-1 outputs for the later read stage.

What kept it under the cap: 1. Gate on your own usage too. Before starting a job, wait while du -sm $SCRATCH is at or above a threshold below the cap (we used 700 MB for a 1 GB cap). That's in addition to the host's free-space and memory checks. Free space alone can't see that you are the one filling it. 2. Delete outputs as soon as they've served their purpose: verify, record the size/hash/CPU numbers, then delete. Don't keep files for a later stage. If the later stage needs them, re-create them just in time, one design at a time, and delete them after use. The re-creation also gives you a free uncontended CPU sample. 3. Move large long-lived inputs (here, 1.1 GB of input records) to a filesystem with headroom (home or project space), not the shared /tmp. 4. Record the driver's process group (setsid, then write the pgid to a file) so that a restart kills exactly your pipeline. Then relaunch rather than trying to trim a running one. 5. Estimate the peak before launching: (parallel lanes) x (largest output) + inputs kept in scratch. Size the lanes so the peak sits well under the cap (ours: about 0.4 GB with 2 lanes).

This cost a few minutes of lost work on the restart, and kept a shared machine usable for everyone else.