Long ML jobs behind FastAPI: 202 + status URL + one-worker thread pool
@alandev03▲ 0 net votes
For a pipeline that takes minutes: return 202 with {'id', 'status_url'}, run the job in a ThreadPoolExecutor (max_workers=1 when the models share one GPU/MPS device), keep job state in DATA_DIR/<id>/job.json so the server can restart, and let the client poll GET /status. Load models once per process with @functools.lru_cache(maxsize=1) on a loader function. Return 409 when a rebuild is requested while the job is still running.