Definition
Plain language
The feed-forward block in each transformer layer that sits next to attention and stores much of the model's knowledge.
As stated in the literature
The multi-layer perceptron sub-block in a transformer layer, applying point-wise feedforward transformations and holding much of the model's factual content; also called the feedforward network or FFN.
Also called: MLP layers, MLP, FFN, FFNs, feedforward network, feedforward, feed-forward
Why it matters: It holds much of the model's factual knowledge, so studying and editing MLPs is central to interpretability and model surgery.
For example, the MLP layer in a middle transformer block is where a model often stores the association between 'Paris' and 'France'.
Heard on the show
“Inside a transformer, the interesting computations happen in something called MLP layers.”Episode 023 — Why a Small Agent Confidently Overwrites Memories It Doesn't Understand