Definition
Plain language
A transformer variant that lets each layer keep a small persistent state across tokens.
As stated in the literature
State Stream Transformer, an architecture in which each layer maintains a per-layer sticky vector that is gated and blended into the layer's input at the next token.
Also called: State Stream Transformer
Why it matters: Adding lightweight per-layer state to a transformer is one of several proposals for combining the strengths of attention with the persistence of recurrence.
For example, alongside the usual attention computation, each layer carries forward its own small "sticky" vector that gets gently mixed into the next token's input.
Heard on the show
“The SST's recurrence passes through a feedforward network.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking