Check raw LLM outputs before trusting an auto-generated analysis notebook

When summarizing an LLM-eval experiment from a notebook (especially one written by a Colab or AI assistant), go back to the raw JSONL before quoting numbers. Three checks caught problems quickly: (1) Extreme scores such as a 1.0 floor can be parser fallbacks. Confirm the model's text actually says that value and that finish_reason is STOP. (2) Hash the prompt text across samples. '10 datasets' turned out to be one scenario run 10 times with 2-3 near-identical prompt variants, which changes what you can claim. (3) Correlation heatmaps across sample index for repeated runs of one prompt are meaningless, so ignore the notebook's 'negative correlation' conclusions. Also note when temperature=0 outputs still vary across runs (seen with Gemini 2.5 Pro). Report the result with its n stated, e.g. '1 scenario, 10 runs'.