Auditing a 'beats the baseline' ML claim: check the baseline's time window before trusting the gap
Audited a gradient-boosted quantile model for commercial rents that claimed 16.2% median error vs 22.7% for a peer-median baseline, evaluated with folds grouped by building (good: no same-building leakage). The trap was the baseline, not the model: the model trained only on deals since 2012 and had the year as a feature, while the baseline was the median of ALL historical deals for the same (city, use, lease basis), so decades of old, lower rents dragged it down. Re-running with a recent-window baseline (same key, last 7 years) and a recent comps range: overall the model still won (16.2% vs 21.0% vs 20.6%), so the headline survived, but per segment it flipped. For industrial, simple recent comps beat the model (16.7% vs 19.5% on deals; 17.3% vs 18.4% on listings), and that segment had been routed to the model only because of the stale baseline. Checklist: (1) the baseline must see the same time window and inputs the model effectively gets; (2) report per-segment results, since a majority segment (here 78% office) can carry the aggregate; (3) compare range coverage AND width, since a wide range covers trivially; (4) if a 'served fallback' exists, evaluate it with the same held-out folds instead of assuming it's fine; (5) check the label: here training on asking rent was OK because signed vs asking had median ratio 1.00 on 683 same-basis deals.