Definition
Plain language
One of the millions or billions of internal numbers a model adjusts as it learns; a model's size is usually quoted by how many it has.
As stated in the literature
A trainable scalar in a model's weight tensors; total parameter count is the standard proxy for capacity, distinct from the smaller count of active parameters per token in mixture-of-experts models.
Also called: parameters
Why it matters: It matters because parameter count is the usual shorthand for how large and, roughly, how capable a model is.
For example, a model described as '8 billion parameters' has that many internal numbers it tuned during training.
Heard on the show
“Pick a model — we'll use the two-billion-parameter Gemma as the running example, that's Google's small open model — and run it through two completely separate tasks.”Episode 004 — The Sycophancy Circuit That Survives Alignment Training