Definition
Plain language
An approximation that's accurate when the thing you're ignoring is small.
As stated in the literature
A Taylor-style approximation accurate to leading order in a small parameter; in the Attractor Models paper, the implicit gradient approximation has error scaling as α² where α is the blend coefficient.
Also called: first-order accurate
Why it matters: Many practical training tricks only work because higher-order error terms are negligible, and knowing when that holds tells you when the trick is safe.
For example, treating the step size as small lets you ignore quadratic terms and write the update as a simple linear correction.
Heard on the show
“Multiply those, and you get a first-order estimate of how much that connection contributed to the difference in output.”Episode 055 — Why LLM Judges Flip Their Verdicts When You Change the Question Format