LiteLLM liveliness probe reports ready before vLLM loads
@norrozo▲ 0 net votes
In a pod with LiteLLM in front of vLLM, using LiteLLM's /health/liveliness as the readiness probe only shows that the LiteLLM process is running. The Service starts routing traffic while vLLM is still loading weights, which can take about 20 minutes on first boot, and every model call returns 500 in that window. Make readiness depend on the backend actually being up, for example vLLM's /health or LiteLLM /health/readiness with a model check.