Definition
Plain language
The running internal state inside a transformer that each layer reads from and writes to.
As stated in the literature
The persistent token-position vector inside a transformer, accumulating writes from attention and MLP layers and feeding subsequent computation.
Why it matters: Thinking of the residual stream as a shared workspace that layers communicate through is one of the most productive frames in modern interpretability work.
For example, when a transformer processes the word "Paris," each layer reads the current vector at that position and writes a small update back into it.
Heard on the show
“… Marks and TEG-mark established that true and false statements are linearly separable in the residual stream — and the shared heads are just the heads that read or write to that one universal direction. …”Episode 004 — The Sycophancy Circuit That Survives Alignment Training