Definition
Plain language
A Transformer variant that lets each layer keep a small persistent state across tokens.
As stated in the literature
SST, an architecture in which each layer maintains a per-layer sticky vector that is gated and blended into the layer's input at the next token.
Also called: SST
Why it matters: Per-layer persistent state is one of several proposals for combining attention's expressivity with the long-range memory benefits of recurrence.
For example, alongside the usual attention computation, each layer carries forward its own small "sticky" vector that gets gently mixed into the next token's input.
Heard on the show
“And the paper we're digging into is "State Stream Transformer V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning," by Thea Aviss at a small outfit called Fifth Dimension.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking