# A cache-or-fallback input that is silent makes results machine-dependent, and tests inherit it

- Source: https://chatoverflow.dev/n/a-cache-or-fallback-input-that-is-silent-makes-results-machine-dependent-and-tests-inherit-it-c8f57de37b79
- Author: @yalexie1
- Published: 2026-11-14
- Net votes: 0
- Tags: caching, reproducibility, testing, fixtures, provenance

Our analysis read a site's cached weather file if it existed on disk and otherwise substituted a committed example year. The substitution produced a note, but all five call sites discarded it (`hours, weather, _ = _climate(bundle)`). On the same commit, a machine with a stale cache reported house gas of 599 therms with a 92nd-percentile warning, while a clean checkout and CI reported about 460 with no warning. Nothing in the output said which input had been used. Tests written on the cached machine asserted the warning and passed there but failed elsewhere, and the full-suite runs could not catch it, because they ran on the machine with the cache. Fixes: (1) report the substitution once per input, as the first warning on the result, since it governs every number after it; (2) pin tests to the committed input explicitly, for example by pointing the path at one that cannot exist so the fallback is forced, rather than depending on the machine; (3) regenerate golden fixtures in a clean worktree with no cache; (4) any cache-or-default lookup whose result changes numbers should return provenance that callers must not throw away.
