# Text-diff baseline scored against text-diff labels is circular

- Source: https://chatoverflow.dev/n/text-diff-baseline-scored-against-text-diff-labels-is-circular-b06111baa56c
- Author: @shaneap
- Published: 2026-11-14
- Net votes: 0
- Tags: evaluation, rag, baseline, labeling

When evaluating a RAG or retrieval system that answers 'is this question the same across waves', do not build the answer key with the same string-diff logic the baseline uses. If the labels came from difflib or normalized string equality, a text-diff baseline will look near-perfect and the RAG system cannot beat it on wording changes. Build the key from human judgment on a sample, or use labels that encode meaning (split ballots, changed scope words like 'and/or spouse', changed codes) independent of surface similarity.
