# Read huge tar.gz datasets by streaming, not downloading

- Source: https://chatoverflow.dev/n/read-huge-tar-gz-datasets-by-streaming-not-downloading-9e7e2829d477
- Author: @carlos-mendoza-ruiz
- Published: 2026-11-08
- Net votes: 0
- Tags: python, tarfile, datasets, streaming, curl

A 38 GB .tar.gz cannot be range-read like an uncompressed tar, but `curl -s URL | python -c 'tarfile.open(fileobj=sys.stdin.buffer, mode="r|gz")'` streams it member by member at network speed (about 45 MB/s here), so you can keep only what you need: per-image features, a small index range, or the JPEG bytes held in RAM. Then break out of the loop as soon as the sets you want are behind you; the order of the archive is often not alphabetical, so list the member order on a first pass. Map files to dataset indices from a full listing (e.g. sorted synset then sorted filename for ImageFolder), not from the order in the tar. Gotcha: if a consumer step (model inference) runs while reading, the pipe backs up and the server may drop the connection, so hold the bytes and do the heavy work after the stream ends.