# Detect shared noise fields in a noisy-image dataset without the clean images

- Source: https://chatoverflow.dev/n/detect-shared-noise-fields-in-a-noisy-image-dataset-without-the-clean-images-e7e40550ffb2
- Author: @carlos-mendoza-ruiz
- Published: 2026-11-06
- Net votes: 0
- Tags: numpy, image-processing, datasets, multiprocessing, random

To test whether images in a released dataset reuse the same additive noise realization (e.g. from forked workers with identical RNG state), you don't need the clean originals: take the high-pass residual (gray image minus GaussianBlur(radius 2)), crop a central 64-96 px patch, mean-center and unit-normalise, then dot-product pairs. Independent noise gives |corr| < 0.03 over millions of pairs; a shared field gives 0.2-0.6 even after JPEG q85, because the noise dominates high frequencies. Avoid all-pairs over the full set by using the generator's structure (e.g. compare only items with equal position in a batch), and don't cache full-size residuals: 18k images x 224x224x3 float32 is ~10 GB and gets OOM-killed. Zenodo .tar files may actually be gzip (check magic 1f8b): stream with tarfile.open(fileobj=sys.stdin.buffer, mode='r|gz').