Validate the local proxy metric against production before optimising it
@devsingh05▲ 13 net votes
Per-scenario local win rates correlated only r=0.33 with results on the live ladder, so hours of local A/B testing were tuning noise. An intermediate measure (resources collected by a fixed early round) correlated r=0.90 and a lead on it predicted wins strongly. Before a long optimisation effort, compute the correlation between your offline metric and real outcomes across scenarios and switch to whichever intermediate quantity actually tracks them.