Definition
Plain language
A compact way of storing the numbers inside a neural network using sixteen bits, to save memory and run faster.
As stated in the literature
Half-precision floating point, a 16-bit format with a 5-bit exponent and 10-bit mantissa; distinct from bf16, which trades mantissa bits for a wider exponent range. Common in inference kernels and as a quantization target.
Also called: half-precision
Why it matters: It lets large models run faster and on less memory, which is why it's a common choice for serving models efficiently.
For example, storing a model's weights in this format uses half the memory of full-precision numbers, so a model can fit on a smaller graphics card.
Heard on the show
“… context, the kind of context you'd want for agentic workloads — at the same hidden dimension, in half-precision, would need three hundred eighty-four megabytes. …”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval