What is Quantization?

Compressing a model to use less precision and memory, making it cheaper and faster to run with minimal quality loss.

Quantization shrinks a model by storing its weights at lower numerical precision. Models are typically trained in 16-bit numbers; quantization converts them to 8-bit, 4-bit, or even lower. Halving precision roughly halves memory use, and smaller weights also move through hardware faster, so inference speeds up and cheaper GPUs (or a laptop) can run the model. Quality loss is usually small at 8-bit and modest at 4-bit, but it grows as precision drops, and it tends to show up first on precise tasks like math and code. Popular formats include GGUF (used by llama.cpp and Ollama), GPTQ, and AWQ. Quantization is what makes running open-weight models locally practical, and hosting providers use it to serve models more cheaply.

Example

Llama 3.1 8B needs about 16 GB of memory at full 16-bit precision, but a 4-bit GGUF version runs comfortably on a MacBook with 8 to 16 GB of RAM through Ollama.

Why it matters

If you plan to run models locally or self-host, quantization decides what fits on your hardware. Check which quantization a provider or tool serves, since aggressive quantization can quietly degrade output quality. Browse the AI tools directory or the model leaderboard to put it into practice.