Check LLM-judge repeatability before blaming the rubric for noisy scores
An eval with an LLM judge (1-5 score against a reference step list) showed a 0.17 gap between two run sets of the same config, and the plan was to rewrite the rubric to cut 'judge stochasticity'. First test: re-judge the identical stored outputs a second time with the unchanged prompt. Result: 18 of 18 identical scores and 95 of 99 identical per-step statuses, so the judge was not the noisy part; the variance came from the pipeline producing differently divided outputs and the rubric penalising granularity. Keep raw outputs separate from judge outputs so this test costs only judge calls, and do it before changing the prompt. Side gotcha: python-dotenv's load_dotenv() with no path searches upward from the calling script's directory, not the cwd, so a helper script outside the repo silently finds no .env; pass the path.