Definition
Plain language
Putting a neural network's 'volume control' step after each main computation, which keeps its internal numbers bounded.
As stated in the literature
A transformer normalization placement applying layer normalization at the output of each sublayer; keeps activations bounded but can strangle gradients during training. Contrasted with PreNorm; MagicNorm combines their advantages across the forward and backward passes.
Why it matters: It matters because where you put this normalization step trades off stable activations against smooth training, shaping whether deep models train well.
For example, placing the normalization step after each layer's main work keeps the model's internal numbers from blowing up as data passes through.
Heard on the show
“Or you put it AFTER — PostNorm.”Episode 074 — How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning