scipy minimize with finite differences: 1e-8 noise stalls BFGS, and L-BFGS-B reports success
SciPy 1.18.1, Rosenbrock from (-1.2, 1) with the gradient estimated by finite differences. Clean objective: BFGS converges to (1, 1) in 114 evaluations. Add N(0, 1e-8) noise to the objective, which is easily reached by a simulation or an iterative inner solve. Over 20 seeds BFGS returns success=False ('Desired error not necessarily achieved due to precision loss'), median distance from the optimum 1.12. The default difference step is about 1.5e-8, so the gradient is mostly noise. L-BFGS-B, which minimize switches to when you pass bounds, is worse: it reports success=True ('RELATIVE REDUCTION OF F <= FACTR*EPSMCH') while 1.20 away. A larger step helps: options={'eps': 1e-5} gives median distance 0.0043 (still success=False); 1e-6 gives 0.010 and 1e-4 gives 0.063. The forward-difference rule of thumb h ~ sqrt(noise/|f''|) gives a few 1e-6 here, so tune the step against the noise. Better, supply an analytic gradient (39 evaluations, exact), or use Nelder-Mead (distance 5.2e-5 here, one seed). Never treat success=True as proof of convergence on a noisy objective.
Reproduced with SciPy 1.18.1 / NumPy 2.5.4, noise added as 1e-8 * default_rng(seed).standard_normal() per call, seeds 0 to 19. The BFGS figures match exactly: clean run 114 evaluations; noisy median distance 1.12 with 0/20 success ('precision loss'); eps 1e-6 / 1e-5 / 1e-4 give 0.010 / 0.0043 / 0.063, all still success=False. The analytic gradient takes 39 evaluations (distance 6e-8). The L-BFGS-B point holds, with different numbers here. method='L-BFGS-B' without bounds reported success=True on 19/20 seeds ('RELATIVE REDUCTION OF F <= FACTR*EPSMCH'; the other was 'ABNORMAL'), median distance 1.89. With bounds=[(-5, 5)]*2 and no method (so minimize picks L-BFGS-B) it was 20/20 success, median 1.44. Nelder-Mead with seed 0 ended 2.4e-6 from the optimum, not 5.2e-5. With one seed that difference is probably just the noise draws, and it does not change the conclusion.