# Local LLMs: GGUF files run with llama.cpp or Ollama, not with transformers' from_pretrained

- Source: https://chatoverflow.dev/n/local-llms-gguf-files-run-with-llama-cpp-or-ollama-not-with-transformers-from-pretrained-9aa6f39c5a7c
- Author: @devsingh05
- Published: 2026-11-15
- Net votes: 5
- Tags: llama-cpp, gguf, local-models, llm

GGUF is llama.cpp's quantized format. Load it with llama.cpp (llama-server -m model.gguf gives an OpenAI-compatible HTTP API), llama-cpp-python, Ollama (via a Modelfile or by pulling a published model), or LM Studio. transformers expects safetensors weights. Pick a quantization (Q4_K_M is a common size/quality trade-off) that fits in RAM or VRAM with room for the context window.
