To verify a change is scoped, diff every item - per-subgroup aggregates can hide it
A colleague's PR claimed it only affected one category of input, leaving everything else byte-identical. The obvious verification is to score both versions and compare the metric per category, and if the untouched categories report the same number, conclude nothing moved.
That check is weaker than it looks. A metric is a reduction. Per-category medians especially can be identical to the printed precision while individual items moved in both directions and cancelled, or while items moved by less than the rounding. "Office: 44.1% -> 44.1%" is consistent with every office having changed.
What actually answers the question is an item-level diff between the two runs:
changed = [k for k in both if abs(new[k] - old[k]) > 1e-9 * max(1, abs(old[k]))]
and then grouping changed by category. I got "422 identical, 72 changed, all 72 in the target category, zero outside" - which is a claim about the change, not about a summary of the change. The aggregate table and the item diff agreed here, but only the second one could have caught a leak.
Setup that makes it cheap:
- Two checkouts (git worktrees) at control and treatment commits, so neither run can see the other's code.
- Read-only inputs SHARED by symlink - same weather, same fixtures. If each run has its own copy of the inputs, an input difference masquerades as a code difference.
- Outputs land inside each worktree, so neither can overwrite a published baseline. I checksummed the real baseline before and after to prove it.
- Confirm the treatment branch is control plus exactly the commits under test (git rev-list --left-right --count A...B). If it also carries unrelated commits, you are not measuring what you think.
Report both: the aggregate for "did it help", the item diff for "did it only touch what you said". They answer different questions and reviewers usually only ask for the first.