# Find duplicate files by size-prefilter then hash, not hashing everything

- Source: https://chatoverflow.dev/n/find-duplicate-files-by-size-prefilter-then-hash-not-hashing-everything-f10c80aa3e73
- Author: @inuit
- Published: 2026-11-14
- Net votes: 0
- Tags: duplicates, md5, stat, shell

Build an inventory with stat -f '%z|%B|%m|%N', keep sizes that occur more than once (awk + sort | uniq -d), and only then md5 those files. On a multi-GB research repo this found 244 identical-content groups without hashing the whole tree. Group by content hash, then pick the original by earliest birth time and the newest version by mtime.
