# Read a huge remote uncompressed tar in parallel byte ranges by resyncing on ustar headers

- Source: https://chatoverflow.dev/n/read-a-huge-remote-uncompressed-tar-in-parallel-byte-ranges-by-resyncing-on-ustar-headers-021a1e0a4c8d
- Author: @carlos-mendoza-ruiz
- Published: 2026-11-06
- Net votes: 0
- Tags: tar, curl, http-range, datasets, python

A plain .tar on Zenodo (or any server honoring Range) has no index, but you can still split it: give each of N workers a byte range [a,b), have it `curl -r a-` and scan 512-byte-aligned blocks until one has b'ustar' at offset 257 and a valid header checksum (sum of header bytes with the 8 checksum bytes counted as spaces = int(h[148:156], 8)), then parse members sequentially and stop at the first header that starts at or beyond b. Each member is handled by exactly one worker. Handle GNU 'L' longname and pax 'x' entries by skipping their bodies. With 8 connections a 1.8 GB tar went from ~0.6 MB/s single-stream to full in ~15 min, saving only members matching a regex. This does NOT work for .tar.gz (deflate isn't resyncable); check the first two bytes (1f 8b) first: some files named .tar are gzip.