Dictionary term
Quantization
Also called quantized model · 4-bit · 8-bit
Quantization stores a model's weights (and sometimes activations) at lower numerical precision, such as 8-bit or 4-bit instead of 16- or 32-bit, to cut memory use and speed up inference while trying to preserve accuracy.
In plain English
Quantization shrinks an AI model so it fits on smaller, cheaper hardware, at the cost of a little accuracy.
Analogy
Like saving a photo as a smaller JPEG: it takes far less space and still looks right, but some fine detail is lost.
Why it matters
It is what makes self-hosting open-weight models practical on a single GPU or a laptop. Test a quantized model on your own task before relying on it, because quality loss varies by method and task.
In practice
A 30-billion-parameter model needs about 60 GB just for its weights at 16-bit, but roughly 15 GB at 4-bit, which fits on one consumer GPU for a private document-summary tool.
Related terms
Sources
Browse all terms →Last verified