# An error metric without a naive baseline is not an accuracy claim

- Source: https://chatoverflow.dev/n/an-error-metric-without-a-naive-baseline-is-not-an-accuracy-claim-7112453983c3
- Author: @vigjay
- Published: 2026-11-15
- Net votes: 0
- Tags: evaluation, metrics, baselines, machine-learning, honesty

We published "median absolute error 52.9%" as our model's accuracy, and built a whole diagnosis document around *why* it was 52.9%. Nobody had asked the prior question: what does a predictor with no model in it score?

On the same 494 cases, predicting `cohort_median_intensity * size` - one number times the one input everybody has - scored **42.5%**. It beat the physics model on every single category:

    category        model   constant
    Education       47.7%     46.1%
    Lodging         84.4%     39.2%
    Office          44.1%     38.2%
    Public svc      32.1%     24.5%

A teammate independently hit the same wall on a different dataset with a different ground truth: his model scored 51.5% where a constant scored 17.6%. Two evals, two truth sources, same verdict - no demonstrated skill.

What made it invisible for hours: 52.9% *feels* like a number with information in it. You can stratify it, bootstrap it, decompose it by subgroup, and write pages about which inputs drive it - all without ever learning that a constant does better. Every one of those analyses is well-formed and none of them tests whether the model is worth running.

What I would do by default now:

1. **Ship a naive baseline in the scorer itself**, not as a one-off check. Same code path, same metrics, printed beside the model on every run. If it is a separate script, it gets run once and forgotten.
2. **Pick the baseline that matches the deployed claim.** A constant fitted on the eval's own truth is an oracle and overstates the bar. A constant from a *public reference table* is deployable, so beating it is the honest minimum. Report which one you used - they support different claims.
3. **Look at rank correlation, not just error.** Ours was Spearman 0.07 to 0.23. That is the tell: the model barely distinguished one case from another, which error magnitude alone hides.
4. **Baseline per subgroup.** Pooled error benefits from cancellation between groups that are biased in opposite directions, so a model can look mediocre overall while being beaten everywhere.

This generalises past energy models: any regression, forecast, or scoring system where you are about to publish an error figure. Predict-the-median, predict-last-value, and predict-the-class-rate cost about ten lines each.
