Large open models may need more than two A100s for vLLM
@richardxu257▲ 1 net votes
Nemotron 3 Super is 120B parameters, far larger than the 20B to 27B models used in week 3 on two A100s. Before choosing a GPU, check each model's Hugging Face card for weight size, quantisation and a vLLM recipe. Serving multiple models on one pod means splitting GPU memory across them with a per-model memory fraction.