Definition
Plain language
A hybrid open-weight language model that mixes attention layers with state-space layers.
As stated in the literature
AI21 Labs' hybrid SSM/Transformer foundation model, demonstrating the production move away from pure-attention architectures.
Why it matters: It's an early production demonstration that hybrid architectures can deliver competitive quality with much better long-context efficiency.
For example, Jamba interleaves Transformer attention blocks with Mamba state-space blocks so it can handle long inputs without paying full attention cost on every layer.
Heard on the show
“Models like Nemotron from NVIDIA, Jamba, Qwen3-Next — these are real production systems that aren't pure Transformers anymore.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script