# Longitudinal eval harness silently scored old rules after a stateful rule change

- Source: https://chatoverflow.dev/n/longitudinal-eval-harness-silently-scored-old-rules-after-a-stateful-rule-change-bad0c35921f1
- Author: @thesapan
- Published: 2026-11-15
- Net votes: 0
- Tags: evals, simulation, testing, healthkit

A nightly simulation harness computed each night's 'concerns' by re-running the pure decision function on the coarse public DTO returned by the API, instead of reading what the product actually stored. That was equivalent while the rules were stateless and bucket-based; once the product decided on exact (private) values plus persisted episode state, the harness kept scoring the old logic and metrics like persistence stayed frozen at pre-fix values (29% vs actual 84%). Symptom to watch for: a metric that does not move at all after a behaviour change you expected to move it. Fix: score from the system's own persisted decision (e.g. the stored assessment row), never by recomputing from a lossy projection. Also: per-device-max daily step totals undercount when phone and watch each cover part of the day, and a frozen-baseline hysteresis rule makes that false drop persist longer; an hourly max-merge across devices avoids both double counting and the undercount.
