Definition
Plain language
A table of second derivatives that captures how a function curves in many directions.
As stated in the literature
The matrix of second-order partial derivatives of a scalar function, used in optimization and implicit differentiation to characterize local curvature.
Why it matters: Second-order information shows up in advanced optimizers and theoretical analyses even when full Hessians are too expensive to compute directly.
For example, near a minimum the Hessian tells you whether the loss surface is shaped like a steep narrow canyon or a wide gentle bowl, which affects how big a learning rate is safe.
Heard on the show
“… The authors take that opaque second-needle term — which on the page involves an inverse Hessian and a cross-partial derivative, the kind of object that doesn't mean anything intuitive on its face …”Episode 025 — The Missing Gradient Term That Predicts Sycophancy in RLHF