Definition
Plain language
Swapping a piece of a model's internal state from one run into another to test what that piece does.
As stated in the literature
An interpretability technique that substitutes internal activations from one forward pass into another to test whether a specific component is causally responsible for a behavior.
Why it matters: It's one of the few techniques that lets researchers make causal claims about what specific parts of a neural network actually do.
For example, you can grab the hidden state from the middle of a model running on 'Paris' and paste it into a run on 'Tokyo' to see if the output flips to 'France'.
Heard on the show
“So the authors lean on a technique called activation patching to figure out where, inside the network, the model is doing the work of retrieving these facts.”Episode 037 — Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say