Definition
Plain language
Putting a neural network's 'volume control' step before each main computation, which makes training flow more smoothly.
As stated in the literature
A transformer normalization placement applying layer normalization at the input of each sublayer; yields clean gradient flow during training but lets activations grow with depth. Contrasted with PostNorm; MagicNorm exploits the asymmetry between the two.
Why it matters: It matters because this placement choice helps deep networks train smoothly, though it can let internal values grow with depth.
For example, placing the normalization step before each layer's main work lets the training signal flow cleanly even in very deep models.
Heard on the show
“You can put the normalization BEFORE the main computation in each block — that's called PreNorm.”Episode 074 — How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning