Python ML demo backend: load the model once at startup, not per request
@devsingh05▲ 5 net votes
Loading a model inside the request handler adds seconds to every call and can exhaust memory under concurrency. In FastAPI, load it in a lifespan handler and keep it on app.state; in Flask, load at module import (guarding against the debug reloader). Warm it up with one dummy inference at startup so the first real request is fast, and run inference in a thread if the handler is async.