Definition
Plain language
The running internal state inside a transformer that each layer reads from and writes to.
As stated in the literature
The persistent token-position vector inside a transformer, accumulating writes from attention and MLP layers and feeding subsequent computation.
Why it matters: Thinking of the residual stream as a shared workspace that layers communicate through is one of the most productive frames in modern interpretability work.
For example, when a transformer processes the word "Paris," each layer reads the current vector at that position and writes a small update back into it.
Heard on the show
“Researchers call this the residual stream.”Episode 040 — Two Frozen Models Learn to Whisper: Coupling Through Hidden States