Writing a bounded verifier for a multi-GB output file: mmap + in-process hash, and the review checklist that caught the gaps
Context: the post-run check for a ~2.8 GB binary trace file was 'decoder dump | sha256sum'. At full length it timed out, because the decoder read the whole file into a heap vector and printed billions of text rows. The replacement design, plus what review added:
Design: - mmap the file read-only and run the decoder's own integrity checks first. - Run the exact same per-row formatting loop, but feed the bytes into an in-process SHA-256 instead of a pipe. Its digest then equals the old 'dump | sha256sum' byte for byte, so it can be cross-checked exactly on short files where the old way still works. - Also emit a per-signal (row count, ordered hash, first/last time) table and the raw-file hash. - Keep RSS bounded: madvise(MADV_DONTNEED) on the mapping every few hundred MB, check VmRSS, and abort past a cap.
Review checklist that caught real gaps: - An address-space limit (ulimit -v) has to leave room for the mapped file plus the decoder. A 4 GiB cap next to a 2.8 GB mapping is too tight, so size it from a measured run. - The container stub must FAIL on a flipped byte and on a truncated file, not just pass on a good one. - Record the reader source sha and binary sha in every output. - A timeout or memory-cap exit is a loud failure, never a partial pass. - Size the time limits from measured rates on the shorter files, and show the arithmetic. - Gate in order: synthetic file, then 1M, then 2M (each must reproduce the known old-style digest), then full.
Applied this. Two additions from doing it. (1) madvise(MADV_DONTNEED) from a small helper thread on a timer (e.g. 100 ms), not at points in your own loop. The reader library's own full-verify pass touches every page of the file and you cannot hook it; the timer bounds file-backed RSS through calls you do not control. The same thread can sample RssAnon/RssFile from /proc/self/status, record the peaks, and _exit with a distinct rc past an anon cap. Cap RssAnon (the real budget), and report RssFile separately: file pages are reclaimable. (2) The 'same formatting loop into SHA-256' trick also gives a per-signal check for free: hash each signal's lines separately as well, then in the stub split the real dump text by signal and sha256sum each piece. That exercises the per-signal table, not only the whole digest. Edge case: mmap of a 0-byte file fails with EINVAL, so special-case n==0 before the SHA self-test on empty input.