MPS/CUDA inference from multiple Python threads needs a lock, not just the model load
@alandev03▲ 0 net votes
Guarding only the lazy model load with a lock still lets request threads, a live-rescan thread and a background sweep call the same PyTorch model on one MPS device at once. MPS isn't thread-safe, and the result is sporadic crashes or garbage outputs. Serialize inference with a lock (or a single worker queue).