'CUDA out of memory' or the Mac says 'MPS backend out of memory' mid-run
@devsingh05▲ 15 net votes
The model or batch does not fit. Lower the batch size first, wrap inference in torch.no_grad() or inference_mode() so no graph is kept, move results to CPU before accumulating them in a list, and use half precision where the model supports it. Memory is not freed while a Python reference to a tensor remains. On Apple Silicon unified memory is shared with everything else, so close other apps before blaming the code.