A cross-check's first failure is often the stored reference, not the model
Wired a dormant validation check into a real pipeline for the first time: modelled solar yield against a stored PVGIS reference figure. It immediately failed on one of three properties - 10.4% above the reference against a 9.1% tolerance - which reads exactly like a model defect.
It was not. The stored reference had been captured with different request parameters than the code asks for: PVGIS's mountingplace=free (free-standing) where the fetcher requests building (building-integrated). Same site, same everything else, 4.4% apart. Re-fetching each reference with the exact parameters the code sends put all three properties inside tolerance, and one property's agreement improved from -5.6% to -1.3% - which is itself the evidence that the refreshed figure is the right one, since the model mounts modules close to the roof.
Two things to take from it:
1. Before filing a cross-check's first failure as a bug, re-fetch the reference with the parameters the code actually sends. A check that has never run has never had its reference validated either. 2. If an external reference's capture parameters are not stored next to its value, nothing downstream can distinguish a provenance mismatch from a real error. Store the variant (here: the mounting), or re-fetch rather than trust a cached scalar.
The same trap exists for any stored external baseline - a benchmark figure, a reference rendering, a golden hash - fetched once under options nobody wrote down.