Fast verbatim-needle scan of HF parquet: range reads + one joined buffer
To scan Hub converted-parquet for benchmark strings, skip downloading: wrap the file URL in a tiny io.RawIOBase that serves read() with HTTP Range requests and pass it to pq.ParquetFile; with iter_batches(columns=<string-typed columns>) pyarrow fetches only the footer and those column chunks (a 214 MB OpenR1 file -> 71 MB fetched; image/audio columns cost nothing). For normalisation, pyarrow replace_substring_regex was the bottleneck (11 s / 500 MB); instead join each string leaf with a \x01 separator, bytes.translate with a 256-byte table (A-Z->a-z, keep 0-9a-z and \x01, all else incl. UTF-8 multibyte -> space), collapse space runs with a numpy mask (keep = ~sp; keep[1:] |= ~sp[:-1]), and run pyahocorasick once over the whole buffer; map match ends to rows with np.searchsorted on separator positions. Remaining cost is the Aho-Corasick iteration itself (~70 MB/s of normalised text).
Checked only the normalisation step (not the range reads or timings). The space-collapse mask keep = ~sp; keep[1:] |= ~sp[:-1] matched re.sub(rb' +', b' ', ...) on 20000 random byte strings (ASCII, punctuation, tabs/newlines, UTF-8 multibyte, \x01) except for one thing: a leading run of spaces is removed entirely instead of collapsed to one (12365 of 20000 differed, all of them only there). That is harmless for substring matching, but normalise the needles with the same function, and don't expect positions to line up with a regex-normalised copy.