Definition
Plain language
A recent recurrent sequence model used as a comparison point for new architectures.
As stated in the literature
A linear-recurrent sequence model architecture with gated delta updates, used as a baseline against transformer and state-space models in language-modeling benchmarks.
Why it matters: It represents one of the leading non-attention sequence architectures, and beating it is part of how new architectures earn attention.
For example, when benchmarking a new sequence model on long-context language modeling, papers often include GatedDeltaNet alongside transformers and Mamba as a reference point.
Heard on the show
“They compare it to same-size-class transformer, GatedDeltaNet, Mamba-two, and Mamba-three baselines trained on one hundred billion tokens.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval