Definition
Plain language
A training trick that keeps a model's adjustments rolling in a consistent direction, like a ball gathering speed downhill, so learning doesn't stall or zigzag.
As stated in the literature
In gradient-based optimization, a term that accumulates a running average of past gradients and adds it to each update, damping oscillations and accelerating progress along consistent directions; a standard ingredient in optimizers like SGD-with-momentum and Adam, and invoked as one of the neural-net-training disciplines borrowed into prompt/skill optimization.
Why it matters: It smooths out training so learning doesn't stall or zigzag, speeding progress along consistent directions.
For example, if a model's adjustments keep pointing the same way, momentum lets them build up speed like a ball rolling downhill rather than restarting each step.
Heard on the show
“" The authors actually pitch it that way — use HExA to get off the ground, then optionally consolidate with gradient RL once you have momentum.”Episode 186 — How a Frozen Model Went From 2% to 77% on Physics Puzzles — Without Retraining