Model Quantization

Model quantization is the technique of reducing the numerical precision of a neural network's parameters—converting 32-bit or 16-bit floating-point weights to 8-bit, 4-bit, or even lower representations—to shrink model size, reduce memory requirements, and accelerate inference with minimal accuracy loss.

The math is straightforward and the impact is dramatic. A 70-billion-parameter model stored in 16-bit precision requires ~140 GB of memory—far more than any consumer GPU. Quantized to 4-bit precision, the same model fits in ~35 GB, runnable on a high-end desktop GPU. At 2-bit (with careful techniques), it can fit on a laptop. This is what enables open-weight models like Llama and Mistral to run locally on consumer hardware, powering applications that would otherwise require expensive cloud API calls.

Modern quantization techniques have become remarkably sophisticated. Post-training quantization (PTQ) converts already-trained models without retraining. GPTQ, AWQ, and GGUF formats have become standard for distributing quantized models. Quantization-aware training (QAT) trains models to be robust to low precision from the start. Mixed-precision approaches keep critical layers at higher precision while aggressively quantizing others. The result: 4-bit quantized models typically retain 95-99% of the original model's capability on standard benchmarks.

Quantization is a key enabler of the Creator Era in AI. When combined with knowledge distillation (smaller models trained from larger ones) and efficient architectures like Mixture of Experts, quantization makes AI accessible without expensive cloud infrastructure. Solo founders can run capable local models for development. Edge devices can perform sophisticated inference. The democratization of AI isn't just about open weights—it's about making those weights small enough to run anywhere.