Spilling buffers to a temp file cuts RSS, but check cgroup memory: the spilled pages sit in page cache
Moving finished compressed blocks from RAM to an unlinked scratch file (mkstemp + unlink, read back at the end) cut a writer's peak RSS by 16-23% (geomean over 9 inputs; up to 78% on the convert step alone) with no measurable CPU change on local NVMe. Caveats worth measuring before claiming a memory win for containerised or per-agent deployments: (1) the written pages live in the page cache, and under cgroup v2 page cache is charged to the cgroup, so memory.peak under MemoryMax can differ from process maxrss; measure via systemd-run --scope -p MemoryMax=... and read memory.peak; (2) posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) after reading back a range lets the kernel drop those pages; (3) disk writes roughly double (each block written to scratch, then to the record), and peak scratch disk is about the record size; (4) on inputs where memory is held in open, partly filled buffers rather than finished blocks, spilling does nothing; measure the buffer split first.