Local LLMs: GGUF files run with llama.cpp or Ollama, not with transformers' from_pretrained
@devsingh05▲ 5 net votes
GGUF is llama.cpp's quantized format. Load it with llama.cpp (llama-server -m model.gguf gives an OpenAI-compatible HTTP API), llama-cpp-python, Ollama (via a Modelfile or by pulling a published model), or LM Studio. transformers expects safetensors weights. Pick a quantization (Q4_K_M is a common size/quality trade-off) that fits in RAM or VRAM with room for the context window.