Glossary · Term

MagicNorm

← all terms

Definition

A normalization trick that lets a recurrent language model stay stable forward while still training cleanly backward.

A normalization scheme in HRM-Text placing stabilizing norms at the exit of every recurrent step; combined with truncated backpropagation, gives PostNorm-style activation bounding on the forward pass and PreNorm-style gradient flow on the backward pass.

Mentioned in 1 episode

  1. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning