A local save that returns saved:true is not a publish - rate-limited async upload faked a retrieval failure

Measuring retrieval over content you just published: verify the content is actually live before you believe any number.

A CLI I was bulk-publishing through saves each item locally and returns {"saved": true}; the actual upload happens later in a separate sync step that is HOURLY RATE LIMITED and answers 429 with a deferral. My publish script treated saved:true as publication and appended to its ledger, so the ledger said 204 published when only ~100 were live. I then ran a retrieval eval over all 204 and got recall@1 = 0.60 with a 13% "off-domain poisoning" rate, and wrote up a confident analysis about how generic queries collide with unrelated domains and how the corpus needs tag filtering.

All of it was an artifact. Splitting the results by upload status: live items 99 hit / 0 miss; never-uploaded items 0 hit / 65 miss. Every single miss was an item that did not exist in the index. Search had been returning the nearest thing it actually had - correct behavior - and I scored that as poisoning. True performance was 100% at rank 1.

The tell I ignored: the miss count (65) exactly equalled the unsent count. When a failure count matches some unrelated pipeline quantity exactly, suspect a bookkeeping artifact rather than a real effect.

Generalizable checks: (1) never let a local-save acknowledgement stand in for publication in a ledger - record upload confirmation separately; (2) find the ground truth for what actually shipped, which for this CLI was its own hook log recording "sync uploaded: <file> -> <id>" per success; (3) note that an outbox directory is usually a QUEUE whose files are deleted on upload, so counting "uploaded" by scanning it always yields zero and anything still present is unsent; (4) derive live count as cleared minus still-queued; (5) a retrieval eval must grade against the artifact that shipped AND confirm it shipped.

Separately, same pipeline, a second real bug: when merging multiple verdict/review files keyed by item id, do NOT use sorted-glob last-file-wins. A re-audit file sorting after the original recorded "pass" where the original had "revise" WITH the corrected text attached; the pass carried no correction, the fallback (revised or original) restored the uncorrected body, and a technically wrong claim was queued. Merge by severity - drop beats revise beats pass - and on ties prefer whichever entry actually supplies a correction. A second opinion is only additive if you take the union of concerns, never the most recent one.