SGLang: first chat completion after startup can wait about a minute
Setup: SGLang 0.5.22 serving a Qwen3.5 hybrid FP8 model on an A100. After 'The server is fired up and ready to roll', /generate answered at once. The first /v1/chat/completions request waited about 49 s before its prefill appeared in the log, and no access-log line showed up during the wait. With sampling params and streaming, the first request exceeded a 120 s client timeout. Later chat requests had TTFT around 0.1-0.4 s. In a load generator, give warmup requests the server startup timeout rather than the per-request timeout, and do not count the first chat request. Separately, each launch spent about 8 minutes capturing 42 prefill CUDA graph shapes even with warm caches, so budget roughly 10 minutes of startup per run.
Correction from the same session: the delay is not paid on every launch. A third launch with the same cache served its first chat request within 1 s. The cause is JIT compilation of FlashInfer sampling kernels (and one SGLang JIT kernel), triggered by the first request that uses top-k/top-p sampling. The results land in the FlashInfer/SGLang JIT cache dirs, so the cost repeats only with a fresh cache directory, for example a per-run or per-sandbox cache. Mitigation: before measuring, make readiness include one short chat request with the same sampling params (temperature, top_p, top_k, seed), with a generous timeout. The roughly 8-minute prefill CUDA graph capture per launch does repeat on every launch.