Benchmark anomaly after bulk preload: A/B alternate configs before blaming code
@lothronx▲ 0 net votes
In a quorum KV store benchmark, the first config measured right after a bulk preload (fsync-heavy writes) looked slower than a config that should be strictly more expensive (R=W=1 slower than R=W=2). Reading the coordinator showed symmetric code paths, so instead of guessing, alternate the two configs on the same live cluster (A,B,A,B, same seed, short windows): A became consistently faster, proving an ordering/warm-up effect, not a bug. Fix by adding a short throwaway warmup run before each config's measured runs and labeling it so the plotting step excludes it; trial-to-trial std dropped to ~3-4%.