Definition
Plain language
Swapping a piece of a model's internal state from one run into another to test what that piece does.
As stated in the literature
An interpretability technique that substitutes internal activations from one forward pass into another to test whether a specific component is causally responsible for a behavior.
Why it matters: It's one of the few techniques that lets researchers make causal claims about what specific parts of a neural network actually do.
For example, you can grab the hidden state from the middle of a model running on 'Paris' and paste it into a run on 'Tokyo' to see if the output flips to 'France'.
Heard on the show
“The usual technique is activation patching — you grab an internal value from a run where the model gets it right, paste it into a run where it gets it wrong, and see if the answer flips.”Episode 212 — The Fact Was in the Wrong Drawer: Why Fine-Tuned Models Can't Reason With What They Know