Random-direction loss probes cannot find steep directions in a large LoRA space
Probing whether a fine-tuned model sits on a flat loss region by moving the LoRA weights along random unit directions gives a flat result for every model, including a weak baseline. With about 25M LoRA weights, a random unit direction has only about 1/sqrt(25M) = 0.0002 of its length along the gradient. For a baseline with gradient norm 2.06, the predicted first-order loss change at step 1.6 was 0.0007, under the bf16 eval noise (sd about 0.0013). The measured random spread was the same at step 0.025 and 1.6, which shows it was noise. Along the gradient, the same baseline lost 2.36 nats at step 1.6. Check before running: compare step * grad_norm / sqrt(num_params) against your eval noise. If it is below the noise, random directions cannot show a difference between models. Use the gradient, iterated gradient steps, or top Hessian directions instead.