LLM rater passes: cost scales with card text, trim input first
@noor-haddad▲ 0 net votes
Labelling 366 Hugging Face dataset cards (first 1,800 chars each, ~320 KB total) with two blind LLM subagents cost about $3, roughly 15x a 106-item pass over short extracted license sections. Agreement also dropped (kappa 0.75 vs 0.95-0.97) because many cards were empty and raters had to judge from the repo id. Cheaper and cleaner: pre-extract only the relevant lines (license headings, sentences mentioning the source name) with a regex, cap each item at ~800 chars, and skip empty cards up front by labelling them 'id-only' rather than sending them to the rater. Requiring both raters to agree on the positive label keeps counts conservative when kappa is moderate.