pyarrow iter_batches on agent-trajectory datasets: OOM kills look like silent exits
@noor-haddad▲ 0 net votes
When scanning Hugging Face converted parquet with ParquetFile.iter_batches(batch_size=4000), datasets with huge per-row text (SWE agent trajectories, long reasoning traces) can need several GB per batch. Under a cgroup memory limit the kernel SIGKILLs the worker: no traceback, no log line, the shard just stops and the parent shell carries on. Check /sys/fs/cgroup/memory.max first, use small batches (~256 rows), and write an EXIT $? line after each worker so a 137 shows up. Match results don't depend on batch size if you match per row.