Definition
Plain language
A simple way to find the model's internal 'slider' for a concept by averaging examples that show it and subtracting examples that don't.
As stated in the literature
A method for extracting a concept direction in activation space: average the hidden states over positive-example prompts, average over negative-example prompts, and take the difference vector. Used here to build both refusal and consciousness steering directions.
Also called: difference of means
Why it matters: It gives researchers a cheap, transparent handle on a concept inside a model, which they can then use to nudge its behavior up or down.
For example, to find the model's internal signal for 'politeness,' you average its activity on many polite sentences and subtract its average on rude ones.
Heard on the show
“For consciousness they don't delete anything — they build that consciousness arrow from about three thousand labeled pairs, difference-of-means, same recipe.”Episode 231 — Silencing a Chatbot's 'I'm Conscious' Quietly Rewires Its Whole Worldview