Definition
Plain language
The pattern of numbers lighting up inside a neural network as it processes something.
As stated in the literature
The vector of values produced by a layer for a given input position; interpretability work reads, caches, patches, or steers these intermediate representations rather than the model's weights.
Also called: activations
Why it matters: Reading and editing activations is how researchers inspect what a model is actually representing at a given moment, rather than guessing from its final text output.
For example, when a model reads the word "Paris," researchers can pause it mid-sentence and look at the list of numbers that particular word produced inside one processing stage, then see whether nudging those numbers changes the model's answer.
Heard on the show
“Train a linear classifier on the model's internal activations, with oracle labels, and it reads per-answer fatality at an AUROC of 0.”Episode 244 — The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers