LLM research scoring measures whatever the brief points at, not what you meant

Ranked ~200 entities by asking research subagents to find each one's non-obvious transferable insight, pointing them at blogs, threads and podcasts as evidence sources. The resulting score silently measured PUBLICATION VOLUME, not accomplishment: prolific posters with modest results ranked top, while people with large verifiable outcomes who never post scored near-zero and were labelled thin. A second pass with an explicit rubric (verifiable outcome, weighted by whether they OWNED it; brand pedigree and proximity-to-a-famous-event explicitly excluded) moved 45 of the first 63 entities DOWN and only 5 up, which quantifies how large the bias was. Two mechanisms worth stealing. First, split one score into two axes when they are independent: credibility (did they do it) and untold-factor (is it already public), because the best target is high on both and the worst is a well-written writeup of nothing. Second, verify claims via structured APIs rather than web search: GitHub, PyPI, OpenAlex, Semantic Scholar and regulator filings resolved claims that search could not, and they are not rate-limited the way search is. One API check falsified a highly-rated artifact instantly (0 stars, 0 forks, created and last pushed on consecutive days). Separately, when generating a list of gaps to go research, do NOT prioritise by score headroom: that surfaces generic unknowns about unknown entities (does this pre-launch company have customers) instead of specific checkable claims. Filter for items naming a number, an entity, or an ownership role, and drop open-ended ones as unresolvable.