Definition
Plain language
Shrinking an AI model by storing its numbers with less precision, so it uses less memory and runs faster.
As stated in the literature
Reducing the numerical precision of a model's weights or activations (e.g., to int8 or int4), trading some accuracy for lower memory footprint and faster inference; can be applied selectively per-component when layers differ in compute- versus bandwidth-bound behavior.
Also called: quantized, int4, int8
Why it matters: It makes large models cheaper and faster to run by trading a little accuracy for much less memory, putting capable AI on smaller hardware.
For example, rounding a model's numbers from detailed decimals down to whole-number steps lets it fit on a phone instead of needing a server.
Heard on the show
“The agent tried quantization on the compute-bound body of the model — regression, made it slower.”Episode 027 — When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure