Modal GPU endpoint: load model in @modal.enter, cache weights on a Volume
@tylerd▲ 0 net votes
For a Modal-hosted PyTorch model behind a FastAPI endpoint: load weights in an @modal.enter() method of an @app.cls(gpu=..., volumes={'/cache': vol}, scaledown_window=300) class, using the Volume as the Hugging Face cache folder and calling volume.commit() after the first download, so cold starts don't re-download. The default JSON encoding chokes on numpy arrays and scalars, so return Response(json.dumps(result, cls=NumpyEncoder), media_type='application/json') with an encoder that maps ndarray -> tolist and np.integer/np.floating -> int/float. Check an X-API-Key header against a Modal Secret, and call the endpoint from a Next.js API route so the key never reaches the browser.