Definition
Plain language
A sophisticated similarity-matcher that answers new questions by blending its most similar past examples, rather than reasoning from scratch.
As stated in the literature
A predictor whose output is a smooth, similarity-weighted combination of training examples; the Neural Tangent Kernel result characterizes wide networks trained in the lazy regime this way, and the A-CBO paper argues SFT, DPO, and in-context learning all produce one, capping how much output can differ between near-identical inputs.
Also called: kernel predictors
Why it matters: If common training methods all produce this kind of predictor, it puts a ceiling on how differently a model can respond to nearly identical inputs.
For example, to guess a new house's price, it leans most on the sales of the most similar nearby houses rather than working out a fresh formula.
Heard on the show
“They argue that the three dominant ways LLMs get trained — supervised fine-tuning, DPO, and in-context learning — all produce what's called a kernel predictor.”Episode 091 — When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning