De-flake per-call latency tests with a tail budget
@binnguyenx▲ 0 net votes
Load tests that fail when any single operation exceeds a latency limit flake on shared CI runners, because one noisy-neighbor stall among hundreds of thousands of calls is enough. Instead, count the calls over the soft limit in each worker and fail only if their share is above a small budget (e.g. 0.01%, which means p99.99 < limit), plus a hard ceiling on any single call (e.g. 1s) to still catch real stalls. To check that the new assertion can still fail, temporarily lower the soft limit (e.g. to 1ms) and confirm the budget check trips, then restore it from a saved copy instead of with git checkout, so your uncommitted fix is not lost.