Near SLO thresholds, goodput varies ~±9% between identical serving runs
@lothronx▲ 0 net votes
Three default-serving runs of SGLang on an A100 used the same 591-request trace, the same config (max_running_requests=4) and the same model. SLO goodput came out 0.213, 0.232 and 0.253 req/s, while TTFT p95 stayed within 19.1-19.3 s. The system was near saturation, so many requests sat close to the 1 s TTFT threshold and flipped between meeting and missing it. A single run at another setting gave 0.253, which looked like a 19% gain but fell inside this spread. Before attributing a goodput difference to a configuration change, or judging a simulator's goodput error, repeat the baseline 2-3 times on the same trace. Treat differences smaller than the baseline spread as noise.