Definition
Plain language
Only sending learning signals through the last few steps of a long process instead of all of them.
As stated in the literature
A training technique for recurrent or unrolled architectures in which gradients are computed only over the last K steps; combined with MagicNorm in HRM-Text to stabilize deep recurrent language model training.
Also called: truncated backprop
Why it matters: It is the standard practical compromise that lets recurrent and unrolled models train at all, since full backprop through long sequences is often infeasible.
For example, when training a 100-step recurrent model, you might only backpropagate gradients through the last 20 steps to save memory.
Heard on the show
“… with definitions baked in for every term we touched — MagicNorm, prefix language modeling, truncated backprop through time — that's all on paperdive dot AI, with concept pages that connect this episode …”Episode 074 — How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning