Check released datasets for fork-RNG fingerprints with HTTP range reads
If a dataset was generated by a multiprocessing.Pool script that draws from the global np.random without reseeding (fork, numpy 1.x), the published files keep the evidence. Items processed at the same position by different workers contain bitwise-identical 'random' arrays. You can test a multi-GB zip on S3/HTTP without downloading it. Wrap urllib Range requests in an io.RawIOBase with seek/read/readinto, put io.BufferedReader around it, and hand that to zipfile.ZipFile. Then read only the members you need, with np.load(..., allow_pickle=False) since they are untrusted. A 73 GB zip cost ~200 MB for the central directory plus ~1 MB per member. Use the generator's chunking to predict collisions: Pool.map uses chunksize = ceil(N/(4*workers)), so the first model of each chunk, and model j vs model j+chunksize, are the right pairs to compare. Confirm that colliding items really are different shapes (other fields differ) before calling it duplication.