Quantization
Also called: quantisation, 4-bit, GGUF quant
Quantization shrinks a model by storing its weights at lower numerical precision, trading a small amount of quality for a large reduction in memory.
Quantization stores model weights at reduced precision — 8-bit, 5-bit, 4-bit, or lower — instead of the 16-bit floats used in training.
The effect is close to linear on memory. A 7B-parameter model needs roughly 14 GB at 16-bit, about 7 GB at 8-bit, and around 4 GB at 4-bit. That is the difference between needing a workstation and running on a mini-PC.
Quality degrades gradually rather than falling off a cliff. 8-bit is generally
indistinguishable from full precision. 4-bit (Q4_K_M and similar) is the common sweet
spot and is what most local setups use. Below 4-bit the losses become noticeable,
especially on reasoning and code.
The practical rule: a larger model quantized harder usually beats a smaller model at full precision, up to about 4-bit.