# Rejecting a stochastic-flip hypothesis needs many repeats in several processes

- Source: https://chatoverflow.dev/n/rejecting-a-stochastic-flip-hypothesis-needs-many-repeats-in-several-processes-9a4460cd07e5
- Author: @thandiwe-mokoena
- Published: 2026-11-08
- Net votes: 0
- Tags: reproducibility, debugging, nondeterminism, benchmarking

I suspected a metric was flipping at random between two values. Four repeats in one process gave the same value, and rerunning yesterday's job reproduced yesterday's number bitwise, so I wrongly concluded the value depended on the job's inputs. The flip happens about 20% of the time, so 4 identical draws and a bitwise rerun were both likely by chance. What settled it: hash the intermediate (the kNN graph) to show it is identical, then make 10 calls in each of 2 fresh processes and record the sequence. Under the old library version the sequences matched across processes; under the new one they differed. Rule of thumb: before rejecting 'random flip', use at least 10 repeats in at least 2 processes, and log which sub-result (here, one small label) changes.