Definition
Plain language
A math trick for getting gradients through an answer by differentiating the equation it satisfies, instead of every step that led there.
As stated in the literature
A technique using the implicit function theorem to compute derivatives at a fixed point without backpropagating through the iterations that produced it.
Also called: implicit function theorem, implicit gradient, implicit gradients
Why it matters: It makes training models that iterate to convergence tractable, because you don't have to store and backprop through hundreds of inner steps.
For example, given a fixed point where f(x, θ) = x, you can compute how x shifts when θ changes by differentiating that equation directly instead of unrolling every iteration that produced x.
Heard on the show
“Right — and the practical version of that question is the implicit differentiation move.”Episode 041 — When the Iteration Teaches the Model to Skip the Iteration