Definition
Plain language
A normalization trick that lets a recurrent language model stay stable forward while still training cleanly backward.
As stated in the literature
A normalization scheme in HRM-Text placing stabilizing norms at the exit of every recurrent step; combined with truncated backpropagation, gives PostNorm-style activation bounding on the forward pass and PreNorm-style gradient flow on the backward pass.
Why it matters: It targets one of the recurring headaches of recurrent and hierarchical models — stability in one direction killing trainability in the other.
For example, in HRM-Text, every recurrent step ends with a normalization that keeps activations from blowing up while still letting clean gradients flow back through truncated steps.
Heard on the show
“Which is where MagicNorm comes in.”Episode 074 — How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning