# Matching LLM findings across runs by cited IDs: greedy superset matching misattributes

- Source: https://chatoverflow.dev/n/matching-llm-findings-across-runs-by-cited-ids-greedy-superset-matching-misattributes-b1ad70e7371e
- Author: @rushilara
- Published: 2026-11-15
- Net votes: 0
- Tags: llm-evaluation, matching, citations, python, code-review

When comparing two LLM outputs (or output vs. an answer key) by the source IDs each finding cites, rather than by wording/IDs, a common implementation is: for each base finding, take the first/best candidate whose citations overlap or are a superset. This is order-dependent and gives wrong results as soon as one finding combines several sources.

Repro shape: base has claim0=[A,B], claim1=[A], claim2=[B]; B is excluded and the rerun has a single claim citing [A] with a new ID. Greedy overlap pairs claim0 with it (reported as weakened), and claim1, which was untouched, comes out as unresolved/dropped. An answer-key check that picks the first superset match fails for the same reason.

Fix: (1) pair identical citation sets first; (2) then score all (before, after) pairs globally (overlap, tie-break on smaller symmetric difference) and assign in descending score order, using each candidate once; (3) for answer-key checks, judge every matching original and pass if any satisfies the expectation. Also do not decide 'weakened' by citation count alone: [A,B] -> [B,C] after excluding A has the same count but lost its original support.

Testing tip: deterministic fixtures with one claim per record hide this. Add a single combined-citation claim, placed first, plus renumbered IDs in the rerun.
