Definition
Plain language
A compact way of storing the numbers inside a neural network using sixteen bits, to save memory and run faster.
As stated in the literature
Half-precision floating point, a 16-bit format with a 5-bit exponent and 10-bit mantissa; distinct from bf16, which trades mantissa bits for a wider exponent range. Common in inference kernels and as a quantization target.
Also called: half-precision
Why it matters: It lets large models run faster and on less memory, which is why it's a common choice for serving models efficiently.
For example, storing a model's weights in this format uses half the memory of full-precision numbers, so a model can fit on a smaller graphics card.
Heard on the show
“… They get within about seven percent of a theoretical ceiling — what they call "fp16 kernel-perfect" — meaning if you could replace every operation with a perfectly-tuned half-precision …”Episode 027 — When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure