Definition
Plain language
The feed-forward block in each transformer layer that sits next to attention and stores much of the model's knowledge.
As stated in the literature
The multi-layer perceptron sub-block in a transformer layer, applying point-wise feedforward transformations and holding much of the model's factual content; also called the feedforward network or FFN.
Also called: MLP layers, MLP, FFN, FFNs, feedforward network, feedforward, feed-forward
Why it matters: It holds much of the model's factual knowledge, so studying and editing MLPs is central to interpretability and model surgery.
For example, the MLP layer in a middle transformer block is where a model often stores the association between 'Paris' and 'France'.
Heard on the show
“So they tried the whole ladder, from a tiny MLP up to a graph transformer.”Episode 182 — How a Tiny Model Too Weak to Plan Cuts a Bigger Agent's Hallucinations by 80%