Counting duplicate/repeated support tickets: lexical cosine undercounts, single-link over-merges

Task: cluster a corpus of support tickets (Discord forum threads) to count how many are repeats of the same underlying issue. No sklearn available; did TF-IDF + union-find by hand.

Two failure modes that matter: 1) Pure lexical TF-IDF cosine (even at a modest 0.34 threshold) UNDERCOUNTS badly. The same root cause is described in totally different words ("Error when aligning a book" vs "container crashes when trying to align"), so cosine misses semantic duplicates. Sweeping the threshold 0.34->0.50 collapsed repeats from ~13 to ~3 tickets — clearly too low. 2) Adding a distinctive-error-token boost edge (union two tickets if they share a rare token like split_tracks / randomuuid / oidc / sycl AND have >=4 shared terms) catches real repeats, BUT single-link union-find then CHAINS unrelated tickets together (A~B, B~C => A,B,C merged) and inflates a couple of giant bogus clusters.

What actually gave a defensible number: report a RANGE and reconcile three methods — strict cosine (floor), boosted single-link (middle), and a hand-curated regex signature per known root cause (interpretable/auditable). The curated signature approach (one regex per recurring root cause, count tickets in any group of >=2) was the most trustworthy and explainable to a stakeholder, because each cluster is human-verifiable. Lesson: for dup-detection over short, jargon-heavy problem statements, curated signatures beat unsupervised clustering, and always publish the methodology + threshold sensitivity, not a single number.