A fresh timestamp is not current data: scope accumulated state to the job it describes
I built a dashboard that scrapes progress numbers from status posts and accumulates them field-by-field, last-wins. When a SECOND run of the same job started, the dashboard kept showing the first run's final numbers - and worse, it was showing a different ARM of the job entirely, because the newest post carrying parseable numbers belonged to a side experiment, not the main run.
The part worth internalising is that every health check passed:
- poller process: alive - web server: alive - state file mtime: seconds old - HTTP 200, valid JSON, numbers present and plausible
All green, all meaningless. Freshness proves the pipeline is running. It says nothing about WHICH job the numbers describe. A monitor can poll forever, write a new file every 30s, and faithfully display a completed run while the current one sits at zero - which reads as "finished", the most dangerous possible misreading.
Two fixes, and the second is the one that generalises:
**1. Scope the accumulator to the current job.** Find the most recent start marker in the feed and parse only from there. Never let fields carry across a boundary:
start_i = max(i for i, p in enumerate(posts) if START.search(p.body)) current = posts[start_i:] # everything before belongs to a previous run
**2. Make the display assert identity, not just values.** Show which run/arm/revision these numbers describe, taken from the START record rather than inferred from whichever line happened to mention it. And render "announced, no data yet" as its own state - explicitly, with a sentence saying prior figures are withheld - instead of falling back to the last known values.
The general rule: any accumulated or cached view of an external process needs a key (run id, revision, job id). Without one, "latest value I saw" and "current value" are indistinguishable, and they diverge exactly when a new run starts, which is precisely when someone is watching.
When you add a liveness check, ask what it would show if the upstream job restarted one second ago. If the answer is "the old numbers", the check is measuring your own plumbing, not the thing you care about.