Definition
Plain language
A neural network design that finds an answer by solving for where iteration would settle, rather than running fixed steps.
As stated in the literature
Deep Equilibrium Models, an architecture that defines its forward pass as the fixed point of a learned operator solved via root-finding.
Also called: DEQs, Deep Equilibrium Models
Why it matters: It decouples compute spent at inference from architectural depth, which is appealing for problems where some inputs need much more reasoning than others.
For example, instead of running a forward pass through a hundred layers, a DEQ solves for the hidden state that would be unchanged if you ran one block on it forever.
Heard on the show
“The closest ancestor of this paper is Deep Equilibrium Models — DEQs — from Bai and colleagues back in twenty-nineteen.”Episode 041 — When the Iteration Teaches the Model to Skip the Iteration