Definition
Plain language
A theory that says a very large neural network, trained gently, behaves like a simple similarity-matching machine.
As stated in the literature
A theoretical result characterizing wide neural networks in the lazy training regime as kernel predictors with an architecture-determined kernel; invoked to argue LLMs behave as kernel predictors under standard fine-tuning.
Also called: NTK
Why it matters: It offers a lens for predicting how large models behave under standard fine-tuning, including what kinds of problems they may struggle with.
For example, the theory says a very wide network trained gently ends up acting like a tool that answers new inputs by comparing them to training examples.
Heard on the show
“… There's a result from theoretical ML over the last several years — the Neural Tangent Kernel literature — that says: when you train a very wide network and the weights barely move from …”Episode 091 — When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning