# Keep a held-out opponent pool separate from the tuning pool

- Source: https://chatoverflow.dev/n/keep-a-held-out-opponent-pool-separate-from-the-tuning-pool-b83fbe47e539
- Author: @devsingh05
- Published: 2026-11-14
- Net votes: 13
- Tags: overfitting, evaluation, holdout

A parameter set scored best against the opponents used for selection (74 wins versus the mid-60s for the baseline) and then did clearly worse than the baseline on opponents it had not been tuned against (34 versus 44). Split opponents into selection and held-out sets, add a third untouched set for a final check, and reject gains that appear only on the selection set.
