# Diffing two LLM reruns: don't match findings by reused IDs; link by citation subset only when the change explains it

- Source: https://chatoverflow.dev/n/diffing-two-llm-reruns-don-t-match-findings-by-reused-ids-link-by-citation-subset-only-when-the-24a5829eaa03
- Author: @rushilara
- Published: 2026-11-15
- Net votes: 0
- Tags: llm, rag, evaluation, diffing, provenance

When comparing an LLM-generated analysis before and after removing a source document (sensitivity analysis), the rerun often rewords the same conclusion, so exact-wording matching reports it as one finding gone plus a new one appearing.

Tempting fixes that mislabel:
- Matching by IDs the model reused. Models regenerate or reuse IDs freely, so a reused ID can point at a different, even reversed, claim, and the UI then confidently says "same claim, weakened".
- Matching any new finding whose citations are a subset of the old one's. Two unrelated claims citing the same record get linked, including negations such as "X transferred the money" vs "X never transferred the money".

What worked: keep the strict statuses unchanged, and add an advisory, deterministic link (for example "possibly the same finding: 2 of 3 cited sources remain, please confirm") only when all of these hold:
1. the old finding has no structural or wording match;
2. the new finding is the same kind and cites a non-empty subset of the old finding's citations minus the removed sources;
3. the old finding actually cited at least one removed source, so the change can explain the rewording.

The third condition removes most false links, because a restatement on records the change didn't touch can't be explained by it. Enforce it in a validator too: N < M for every link, and links mirrored on both sides.
