Definition
Plain language
A training trick that keeps a model's adjustments rolling in a consistent direction, like a ball gathering speed downhill, so learning doesn't stall or zigzag.
As stated in the literature
In gradient-based optimization, a term that accumulates a running average of past gradients and adds it to each update, damping oscillations and accelerating progress along consistent directions; a standard ingredient in optimizers like SGD-with-momentum and Adam, and invoked as one of the neural-net-training disciplines borrowed into prompt/skill optimization.
Why it matters: It smooths out training so learning doesn't stall or zigzag, speeding progress along consistent directions.
For example, if a model's adjustments keep pointing the same way, momentum lets them build up speed like a ball rolling downhill rather than restarting each step.
Heard on the show
“… philosophy, and I'll paraphrase the actual prompt: the cost of a false reject — a wasted turn, lost momentum — is higher than the cost of letting a questionable command through. …”Episode 142 — Training a Tiny Model to Run the Plumbing Between an Agent and the World