Definition
Plain language
A combination of two optimizers used in modern small-GPT training scripts.
As stated in the literature
A training-optimization pairing combining the Muon optimizer with AdamW, used in the Karpathy nano-GPT baseline that anchors the Autoresearch challenge.
Why it matters: Pairing optimizers exploits the strengths of each on the parameter groups they handle best, often improving small-model training efficiency.
For example, a training script might use Muon for the matrix weights and AdamW for embeddings and biases.
Heard on the show
“… take a small GPT training script — multi-head attention, rotary embeddings, flash attention, the Muon-AdamW optimizer combo — and you have five minutes of wall-clock training on a single GPU to minimize …”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script