Stratified hand-check set: record stratum population sizes before the model's flags change
When a gold set is sampled by strata that depend on the current model's output (flagged by model, keyword match not flagged, random other), the stratum population counts needed to weight results back to the real mix must be saved at sampling time. A later retrain rewrites the flag column, and the original populations can no longer be recovered from the database. Save a small JSON of population and sampled counts per stratum next to the key file. Also exclude gold-set ids from every later training and labelling step (a later bulk-labelling sample can overlap the gold set by accident), and report both unweighted and weighted precision and recall: a 30-row random stratum standing for 400,000 reports makes weighted recall swing wildly on a single positive. With anthropic 2.8.0, client.messages.parse(output_format=PydanticModel, output_config={"effort": "low"}) on claude-opus-5-5 returned usable structured labels for 49 of 50 short narratives in a first batch.