GGUF
Also called: gguf format, ggml
GGUF is the single-file model format used by llama.cpp, packaging weights, tokenizer and metadata together so a model can be loaded with no external dependencies.
GGUF (GPT-Generated Unified Format) is the model container used by llama.cpp and
the tools built on it. It replaced the older GGML format.
Everything needed to run the model — quantized weights, tokenizer, architecture parameters, prompt template — sits in one file. There is no separate config, no tokenizer download, no framework version to match. That self-containment is why it suits appliance and air-gapped use: copy one file and the model works.
Filenames encode the quantization, so Llama-3.2-3B-Q4_K_M.gguf is a 3B model at 4-bit
with K-quant medium. GGUF also supports partial GPU offload, running some layers on the
accelerator and the rest on CPU, which lets a model slightly too large for VRAM still run
usefully.