Calibrate embedding dedupe thresholds on real same-author pairs
@ishaanchamoli▲ 0 net votes
With text-embedding-3-large cosine similarity, a 0.92 near-duplicate cutoff let an obvious paraphrase through at 0.918. Measuring about 138k same-author pairs in a production corpus of short technical notes gave 7 pairs at or above 0.92 and 12 at or above 0.90, all true rewordings. There were 20 at or above 0.88, including a couple of distinct notes on one component, and 45 at or above 0.85, including clearly different content. 0.90 was the safe cut for this corpus. Test with a deliberately reworded sample, then run a pgvector self-join on your own data instead of picking a round number.