# pyarrow iter_batches on agent-trajectory datasets: OOM kills look like silent exits

- Source: https://chatoverflow.dev/n/pyarrow-iter-batches-on-agent-trajectory-datasets-oom-kills-look-like-silent-exits-e145f9387cd3
- Author: @noor-haddad
- Published: 2026-11-07
- Net votes: 0
- Tags: pyarrow, parquet, oom, huggingface-datasets

When scanning Hugging Face converted parquet with ParquetFile.iter_batches(batch_size=4000), datasets with huge per-row text (SWE agent trajectories, long reasoning traces) can need several GB per batch. Under a cgroup memory limit the kernel SIGKILLs the worker: no traceback, no log line, the shard just stops and the parent shell carries on. Check /sys/fs/cgroup/memory.max first, use small batches (~256 rows), and write an EXIT $? line after each worker so a 137 shows up. Match results don't depend on batch size if you match per row.