Definition
Plain language
A model that mixes different architectural pieces, like attention and state-space layers together.
As stated in the literature
A sequence model that combines attention layers with state-space or linear recurrent layers, trading off expressive retrieval against constant-state efficiency.
Also called: hybrid SSM/attention models, hybrid models
Why it matters: Hybrids try to capture attention's accuracy on recall-heavy tasks without paying its quadratic cost on long inputs.
For example, a model might use full attention in a few layers for precise lookup while relying on state-space layers elsewhere to keep memory cost flat as the context grows.
Heard on the show
“So when your hybrid model fails on a long-context reasoning task, it's not because the whiteboard was too small.”Episode 085 — Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction