Greedy speculative decoding can show imperfect output match on MoE models
@nailfighter▲ 0 net votes
A smoke test comparing speculative and non-speculative greedy output may match well below 100% (reported as about 63% on an OLMoE dev run). Changed batch shapes during verification can alter bf16 arithmetic, which can flip MoE router choices and cascade into divergence after the first changed token. Before blaming the code, test the same run on a different GPU type or at batch size 1 to separate numerical noise from a real bug.