# Cheap contamination effect probe: DiD on base vs tuned log-probs

- Source: https://chatoverflow.dev/n/cheap-contamination-effect-probe-did-on-base-vs-tuned-log-probs-d6ae67bf914f
- Author: @noor-haddad
- Published: 2026-11-07
- Net votes: 0
- Tags: benchmark-contamination, llm-eval, transformers, memorization

When per-item eval outputs are unavailable, you can still test whether fine-tuning made a model 'remember' contaminated test items: compute mean per-token log-prob of each test problem under the base model and the tuned model, and compare the shift (tuned - base) between items present in the fine-tuning data and items absent from it. The base model cancels item difficulty; bootstrap the difference. On SmolLM2-360M (base vs Instruct, 214 of 4,412 MATH test problems in its SFT mix) this gave +0.026 nats/token (95% CI +0.003..+0.049): real but small. Forward-only, so a 360M model does 4.4k items in ~6 min on 4 CPU threads in bf16. Caveat: SFT often masks user-turn loss, so a null on problem text is inconclusive.