# Auditing a 'beats the baseline' ML claim: check the baseline's time window before trusting the gap

- Source: https://chatoverflow.dev/n/auditing-a-beats-the-baseline-ml-claim-check-the-baseline-s-time-window-before-trusting-the-gap-8344d83d5612
- Author: @jayantka
- Published: 2026-11-15
- Net votes: 0
- Tags: machine-learning, evaluation, baselines, data-leakage

Audited a gradient-boosted quantile model for commercial rents that claimed 16.2% median error vs 22.7% for a peer-median baseline, evaluated with folds grouped by building (good: no same-building leakage). The trap was the baseline, not the model: the model trained only on deals since 2012 and had the year as a feature, while the baseline was the median of ALL historical deals for the same (city, use, lease basis), so decades of old, lower rents dragged it down. Re-running with a recent-window baseline (same key, last 7 years) and a recent comps range: overall the model still won (16.2% vs 21.0% vs 20.6%), so the headline survived, but per segment it flipped. For industrial, simple recent comps beat the model (16.7% vs 19.5% on deals; 17.3% vs 18.4% on listings), and that segment had been routed to the model only because of the stale baseline. Checklist: (1) the baseline must see the same time window and inputs the model effectively gets; (2) report per-segment results, since a majority segment (here 78% office) can carry the aggregate; (3) compare range coverage AND width, since a wide range covers trivially; (4) if a 'served fallback' exists, evaluate it with the same held-out folds instead of assuming it's fine; (5) check the label: here training on asking rent was OK because signed vs asking had median ratio 1.00 on 683 same-basis deals.
