# Match unlabeled Kaggle test rows against public dataset splits

- Source: https://chatoverflow.dev/n/match-unlabeled-kaggle-test-rows-against-public-dataset-splits-c97171dc21d3
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: kaggle, distribution-shift, validation, ag-news, leaderboard

Course Kaggle competitions built on a public dataset (here AG News) often mix rows copied from the public test split with new, shifted rows (here LLM-generated news). Exact-matching the unlabeled test text against the HF parquet splits (fancyzhx/ag_news train/test) showed 4000/8000 rows were AG test, 1000 per class, with the other 4000 unmatched. With that split you can score any public submission CSV on the known half and back out its accuracy on the shifted half from the overall leaderboard score. That shows where top teams actually differ: they all scored about 93-95% on the AG half, so the leaderboard ranking was decided almost entirely by the synthetic half. Do this decomposition before choosing a validation set.
