Projecting disk exhaustion during a long run: use >=2-min windows, subtract known output growth, and don't trust /proc write_bytes as net growth

Context: a long simulation writing a ~10-12 GB dump to a root filesystem with ~20 GB free, and an in-run abort floor at 8 GB. We needed to know whether it would finish.

What misled us: - An 11-second sample of df showed free space falling at ~6 MB/s while the dump grew at ~3.9 MB/s. That implied an unknown ~2 MB/s writer, and the projection put the run below its abort floor. A 120 s window showed free space falling by exactly as much as the dump grew (460 MB vs 460 MB). The 'extra writer' was sampling noise: filesystem free-space accounting and delayed allocation/writeback make short windows jumpy. - /proc/<pid>/io write_bytes for another process (an editor session with a large SQLite database) showed ~9.7 MB/s. Its net file growth was only ~0.09 MB/s, because write_bytes counts bytes sent to storage, including rewrites of existing pages, not net new space.

Practice: - Measure free-space slope over at least 1-2 minutes (better, a regression over several samples). Compare it with the growth of the files you know about (stat size of the output) over the same window. The residual is the unexplained writer. - To find real space consumers, look at file size or du growth over time, not write_bytes. A sqlite/WAL rewriter can show a huge write rate with almost no net growth. - Put the abort floor and an earlier alert line in the run driver itself, so a disk-full ends the run cleanly with a recorded reason rather than a corrupt dump. - You can't move a file that is being written to another filesystem mid-run. If space is tight, write output to the big volume from the start. - Keep big scratch volumes separate from root for long jobs. Root also holds unrelated state (editor caches, logs) that other users grow.