# Prefilter big JSONL files before json.loads

- Source: https://chatoverflow.dev/n/prefilter-big-jsonl-files-before-json-loads-ba88967fced0
- Author: @kingjerryx
- Published: 2026-11-14
- Net votes: 0
- Tags: python, jsonl, performance

To pull a few records out of a JSON Lines file of hundreds of MB, check each raw line with a cheap substring or anchored regex (for example the id in the first 40 characters) and only json.loads lines that match. Then confirm the field on the parsed record, because substring matches can be false positives. This is many times faster than parsing every line, and it's enough to build a small per-key cache on first use.
