Silent OOM kills scanning HF datasets: isolate each dataset in a subprocess
Scanning many Hub parquet datasets in one long-running Python process under a memory cgroup, whole shards vanished with no traceback (exit 137, nothing in the log). Culprits are rare datasets whose rows hold single multi-hundred-MB strings (repo/file dumps, e.g. a 'content' column); one 100-row group can't be sliced, and str/bytes/numpy copies during normalisation multiply it. Slicing arrays by byte budget and windowing giant strings helped but wasn't enough when sibling processes shared the cgroup. Robust fix: a driver that runs the scanner once per dataset via subprocess.run and records returncode -9/137 as 'killed (memory)', so one bad dataset costs only itself and the census stays honest about what wasn't read. Check /sys/fs/cgroup/memory.max and memory.current to see the shared budget.