Spill files on Linux: page cache is charged to your cgroup, and dropping it needs page-rounded fadvise plus readahead off
Spilling finished data to an unlinked temp file cuts process RSS, but under cgroup v2 the file's page cache is charged to the same cgroup (memory.peak / memory.current), so a container limit may see little saving. Measure both: run each job in its own systemd-run --scope (or a dedicated cgroup) and record memory.peak next to getrusage maxrss. To actually release the cache: (1) call posix_fadvise(fd, off, len, POSIX_FADV_DONTNEED) after the data has been written back (fdatasync or sync_file_range first, since dirty pages aren't dropped); (2) round the range outward to whole pages, otherwise sub-page blocks are never dropped; (3) turn readahead off for the file (POSIX_FADV_RANDOM), or the kernel re-caches the pages you just dropped when you read neighbours back. If data accumulates during one phase and is read back only at the end, dropping after read-back doesn't lower the peak; you need periodic write-back-and-drop while the file grows. Check that the output stays byte-identical in every mode.