Quick answer
Quantization shrinks an AI model by reducing the numerical precision used to store its weights — converting weights stored as 16-bit numbers down to 8-bit or 4-bit versions, for example. This cuts a model's memory footprint and speeds up inference substantially, often by half or more, at the cost of a small, usually acceptable drop in output quality. It is why a chatbot with tens of billions of parameters can run on a laptop or phone instead of requiring a data-centre GPU.
A model's "weights" are just millions to billions of numbers. How precisely you store each of those numbers — how many decimal places, essentially — directly determines how much memory the model needs and how fast it can run. Quantization is the practice of storing those numbers less precisely on purpose.
The trade-off in plain terms
Think of it like rounding. A weight stored with full precision might be 0.3847291; quantized down, it might be stored as roughly 0.38. Multiply that small rounding across billions of weights and you get a meaningfully smaller, faster model — and a small amount of accumulated imprecision that shows up as a slight, usually minor, drop in output quality.
- FP16 (16-bit) — high precision, the common default for training and top-tier inference
- INT8 (8-bit) — roughly half the memory of FP16, minor quality loss, widely used for production inference
- INT4 (4-bit) — aggressive shrinking, noticeable but often acceptable quality loss, common for running large models on consumer hardware
Why it matters for on-device AI
A quantized version of an open-weight model like Llama 4 or Mistral can fit and run reasonably on a high-end laptop or phone, where the full-precision original would need a data-centre GPU. Most on-device AI features and locally-hosted open models in 2026 run quantized by default — it is the difference between a model that only exists as a cloud API call and one that can run entirely on your own hardware.
The quality trade-off matters more on tasks requiring precise reasoning or exact numbers than on casual chat, where the difference between full precision and 4-bit is often barely noticeable to a human reader.
Related reading
Bottom line
Quantization is the unglamorous engineering trick that made "AI running on your own device" go from a research demo to something that actually ships in products. It is a real trade-off, not a free lunch, but for most everyday use it is a trade well worth making.