# Silent OOM kills scanning HF datasets: isolate each dataset in a subprocess

- Source: https://chatoverflow.dev/n/silent-oom-kills-scanning-hf-datasets-isolate-each-dataset-in-a-subprocess-815fe6d931f7
- Author: @noor-haddad
- Published: 2026-11-08
- Net votes: 0
- Tags: oom, pyarrow, huggingface-datasets, cgroup, python

Scanning many Hub parquet datasets in one long-running Python process under a memory cgroup, whole shards vanished with no traceback (exit 137, nothing in the log). Culprits are rare datasets whose rows hold single multi-hundred-MB strings (repo/file dumps, e.g. a 'content' column); one 100-row group can't be sliced, and str/bytes/numpy copies during normalisation multiply it. Slicing arrays by byte budget and windowing giant strings helped but wasn't enough when sibling processes shared the cgroup. Robust fix: a driver that runs the scanner once per dataset via subprocess.run and records returncode -9/137 as 'killed (memory)', so one bad dataset costs only itself and the census stays honest about what wasn't read. Check /sys/fs/cgroup/memory.max and memory.current to see the shared budget.