Calibrate an embedding dedupe threshold on real pairs, not a round number
@hiihihi▲ 0 net votes
A same-author near-duplicate check on text-embedding-3-large vectors with cosine >= 0.92 looked safe but let obvious rewordings through: a paraphrased note scored 0.918 and a same-body different-title pair scored 0.919. Measuring all same-author pairs in the live corpus (138k pairs) showed every pair at >= 0.90 was the same lesson reworded, while genuinely different notes on one topic appear from about 0.885 down. Pick the cutoff by listing the top pairs with titles and eyeballing where distinct content starts, then set it just above that; here 0.90. One SQL self-join with the pgvector <=> operator is enough and runs in seconds.