Lookup/verification tool that buffers every output row: fine on short inputs, std::bad_alloc on long runs. Measure bytes per row, then choose a bigger box or a streaming checker
Setup: a read-back tool evaluated every requested name over a whole run and printed each history at the end. Short test inputs ran in seconds at a few hundred MB, so its memory use looked like a non-issue.
Symptom: in a corpus sweep run under an address-space cap (ulimit -v 6 GB), every short file passed, and a group of the longest runs (tens of millions of output rows) died with rc 134: std::bad_alloc, uncaught, so terminate/abort. The cap is what made it a clean, attributable failure. Without it the same runs would have pushed a busy shared host into swap.
Finding the cause: the failing files were exactly the ones with the most output rows, not the biggest inputs or the most names. That points at accumulated output, not input loading. The tool keeps every (time, value) row of every history in memory until it prints.
Measuring it: re-run the failing files unchanged on a bigger box, one per job, with a generous cap. Record peak RSS (getrusage maxrss / /usr/bin/time -v) and the output row count per file. Peak RSS grew linearly with rows: about 100 B per row, roughly 5 GB at ~50M rows and ~7.5 GB at ~70M rows. Treat it as a lower bound for the next, longer input, and use it to predict which files need what box before submitting.
Two ways out: 1. Run the shipped tool unchanged on a box sized from bytes-per-row x expected rows, plus headroom. The verdict still comes from the real tool, which matters when the point is "the shipped reader gets these values". The cost is a box per long file. 2. A streaming checker that hashes or compares each history as rows are produced, so memory is O(names), not O(rows). It runs anywhere, but it is no longer the shipped tool. Prove it equal to the shipped tool on the files where both fit before trusting it on the rest.
Also: log peak RSS and row count per file in every sweep. One line per file makes the linear growth obvious before the first abort.