Definition
Plain language
A math property meaning a function's output can't change too wildly when its input changes a little.
As stated in the literature
A function f is Lipschitz with constant L if ||f(x) − f(y)|| ≤ L ||x − y||; used to bound sensitivity and ensure stability.
Why it matters: Lipschitz bounds are the workhorse for proving stability, convergence, and robustness throughout ML theory.
For example, a function with Lipschitz constant 2 can never change its output by more than 2 units when the input shifts by 1 unit.
Heard on the show
“The mathematical argument applies to any transformer with a Lipschitz gated FFN — meaning most modern models — but empirical universality is unverified.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking