Definition
Plain language
A model's internal working memory at a given moment — the numbers it carries between steps.
As stated in the literature
The intermediate vector representation maintained by a neural network at each token position or recurrent step, capturing accumulated context.
Also called: hidden states
Why it matters: Everything a model 'knows' in the moment lives in its hidden state, which is why probing and editing those states is central to interpretability.
For example, after reading 'The cat sat on the,' a model's hidden state encodes enough context to make 'mat' a likely next word.
Heard on the show
“First, the hidden state: as the model processes text, it builds an internal vector at every layer — a long list of numbers that works like a snapshot of its working memory at that instant.”Episode 204 — The Length Estimate Hiding Inside a Word-by-Word Model