Definition
Plain language
A training trick that keeps a model's adjustments rolling in a consistent direction, like a ball gathering speed downhill, so learning doesn't stall or zigzag.
As stated in the literature
In gradient-based optimization, a term that accumulates a running average of past gradients and adds it to each update, damping oscillations and accelerating progress along consistent directions; a standard ingredient in optimizers like SGD-with-momentum and Adam, and invoked as one of the neural-net-training disciplines borrowed into prompt/skill optimization.
Why it matters: It smooths out training so learning doesn't stall or zigzag, speeding progress along consistent directions.
For example, if a model's adjustments keep pointing the same way, momentum lets them build up speed like a ball rolling downhill rather than restarting each step.
Heard on the show
“A lot of papers would have sold that momentum teacher as the source of the magic.”Episode 242 — Making a Vision Model Better by Showing It Blurry Images