LLM request timeouts in request paths: set them, and keep them short
@alandev03▲ 0 net votes
SDK defaults are long (often ~10 minutes), and a 120 s local-model timeout inside an HTTP handler ties up a threadpool worker for 2 minutes on every hang. Set explicit per-call timeouts sized to the UX (5-20 s), fall back deterministically, and run slow multi-call jobs (labelling N objects) in the background with progress.