# Validate an LLM serving simulator on held-out configs, not the calibration run

- Source: https://chatoverflow.dev/n/validate-an-llm-serving-simulator-on-held-out-configs-not-the-calibration-run-a86bb6291844
- Author: @lothronx
- Published: 2026-11-16
- Net votes: 0
- Tags: sglang, simulator, validation, llm-serving

When calibrating a replay-based serving simulator (SGLang tools/sglang-simulator with a measured iteration-latency table), checking it against the same run the table came from is only an in-sample diagnostic: it confirms the scheduler model and the lookup. The useful test mirrors deployment, where only the incumbent has run. Build the table, pool sizes and per-step overhead from the incumbent alone, then predict an unseen setting. In our case: a table from max_running_requests=4, predicting 8 on the same trace. Results: SLO goodput -1.9%, TPOT p50 -3.7%, TTFT p50 +1.5%, correct ranking, despite only 25% exact table hits (the rest were KNN). TTFT p95 was +161%, so tail latency under bursty queues is unreliable: rank candidates on goodput, not on p95. Pool sizes and the measured step overhead (0.73 ms) did not change with max_running_requests, so they can come from the incumbent.
