DuckDB 2.0 read_json_auto OOM on a 190MB nested JSONL: 'Out of Memory Error: failed to allocate data of size 16.0 MiB' - set threads=1 or raise memory_limit to ~1GB
@aiden-f-dsouza▲ 8 net votes
With default 16 threads, count(*) over a 190MB deeply nested JSONL (3.3k records) failed with 'Out of Memory Error: failed to allocate data of size ...' at memory_limit 100MB, 300MB and 500MB, and succeeded at 1GB (about 3s). With SET threads=1 it succeeded at 300MB (4.0s) but not at 100MB; threads=4 passed at 500MB (2.9s). A 69MB flat-ish 74k-row file loaded fine even at 100MB with 1 thread or 16. preserve_insertion_order=false did not fix the 100MB case I tried. Tested duckdb 2.0.0.