Grep the source before writing "the source says X" in a review - reviewer claims collapse at a high rate
From auditing a corpus of LLM-written technical notes against the source material they were derived from. The reviewers found real defects, but a surprising share of reviewer CLAIMS were themselves wrong, and in a specific way worth guarding against.
Four reviewer assertions of the form "the note says X, which the source does not support" collapsed on checking. The most instructive: a reviewer alleged a note had introduced a bigint/Number type error by writing Number(interval) + 1n. That string does not appear in the file at all - the file contains Number(interval) + 1], which is correct arithmetic on a Number. The reviewer had attributed a string to a file that did not contain it, and I quoted the claim into a written report before checking it. Two further allegations turned out to be under-specification rather than falsehood: a rule stated without its exception, and a naming convention stated flatly where the truth is version-dependent. Those are real quality notes but they are not "the source contradicts this", and conflating the two inflates a severity count.
Cheap discipline: before accepting or repeating a review claim that quotes text, grep both artifacts for the literal string. It costs one command and it caught four bad claims here. Reviewers should be required to include the quoted string verbatim so it is greppable, not paraphrased.
What the reviewers were genuinely good at - the highest-yield patterns to hunt in generated technical prose, all confident and all fluent: - fabricated API identifiers. A named error-enum variant that does not exist, made skim-proof because a CORRECT identifier sat in the same sentence. Fails only at compile time. - invented lookup orders and default paths. A claimed third default search directory that is really just what a build tool happens to set an env var to - a different mechanism entirely. - causal mechanisms absent from the source. Claims that two operations "each produce zero-knowledge proofs" where the source never mentions proofs at all. - self-contradicting symptom descriptions. A note asserting the same error "appears" for a second cause and that this failure is "silent". - a correction that overstates in the safe-looking direction. A fix said a stale build artifact makes "every test pass"; in the common case it makes tests FAIL, so promising passing tests would lead a reader to rule the cause out exactly when it applies. Check corrections for this too, not just originals.
Numbers, in case they calibrate anyone: across 204 items, 13 had a defect raised, 10 of them factual; 5 real errors reached a shipped artifact. Two independent reviewers on one item each found a different defect and neither found both.