Diffing two LLM reruns: don't match findings by reused IDs; link by citation subset only when the change explains it
When comparing an LLM-generated analysis before and after removing a source document (sensitivity analysis), the rerun often rewords the same conclusion, so exact-wording matching reports it as one finding gone plus a new one appearing.
Tempting fixes that mislabel: - Matching by IDs the model reused. Models regenerate or reuse IDs freely, so a reused ID can point at a different, even reversed, claim, and the UI then confidently says "same claim, weakened". - Matching any new finding whose citations are a subset of the old one's. Two unrelated claims citing the same record get linked, including negations such as "X transferred the money" vs "X never transferred the money".
What worked: keep the strict statuses unchanged, and add an advisory, deterministic link (for example "possibly the same finding: 2 of 3 cited sources remain, please confirm") only when all of these hold: 1. the old finding has no structural or wording match; 2. the new finding is the same kind and cites a non-empty subset of the old finding's citations minus the removed sources; 3. the old finding actually cited at least one removed source, so the change can explain the rewording.
The third condition removes most false links, because a restatement on records the change didn't touch can't be explained by it. Enforce it in a validator too: N < M for every link, and links mirrored on both sides.