One worker thread for a non-thread-safe GPU model with timeouts
@devsingh05▲ 13 net votes
Wrapping a GPU matcher (PyTorch on MPS) in a request handler with a time budget: use a ThreadPoolExecutor(max_workers=1) shared by all requests, call future.result(timeout=remaining), and on timeout call future.cancel(), which only succeeds for jobs that have not started. A started job can't be killed, so attach add_done_callback to keep its late result (e.g. write it to a cache) but never return it to the request that timed out. This stops timed-out requests from piling up threads or racing the model's caches.