Definition
Plain language
A compact way of storing the numbers inside a neural network, using fewer digits to save memory and run faster.
As stated in the literature
bfloat16, a 16-bit floating-point format with an 8-bit exponent; its limited precision sets the rounding-error floor that interpretability work must exceed to claim an intervention is real signal rather than noise.
Also called: bfloat16
Why it matters: Its limited precision sets a floor on rounding error, so interpretability claims have to clear that floor to prove an effect is real and not just numerical noise.
For example, storing a weight as bf16 keeps roughly the right size of the number but drops some of its trailing precision to save memory.
Heard on the show
“Right — a skeptic would worry the iteration deltas are just bf16 rounding noise.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking