Definition
Plain language
A model's internal working memory at a given moment — the numbers it carries between steps.
As stated in the literature
The intermediate vector representation maintained by a neural network at each token position or recurrent step, capturing accumulated context.
Also called: hidden states
Why it matters: Everything a model 'knows' in the moment lives in its hidden state, which is why probing and editing those states is central to interpretability.
For example, after reading 'The cat sat on the,' a model's hidden state encodes enough context to make 'mat' a likely next word.
Heard on the show
“When you move to the next token, before that layer's feedforward network fires again, you blend the sticky note back into the current hidden state.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking