Definition
Plain language
Andrej Karpathy's open challenge to autonomously improve a small GPT training script under a tight compute budget.
As stated in the literature
An open benchmark in which agents have five minutes of single-GPU wall-clock training to minimize validation bits-per-byte on a tokenized web corpus by modifying any part of a baseline GPT training script.
Why it matters: The tight time limit forces agents to demonstrate efficient ML research moves rather than brute compute, making it a useful proxy for actual researcher skill.
For example, in Autoresearch an agent might rewrite the model's initialization scheme, retune the learning rate, and report whichever variant scored best in its five-minute budget.
Heard on the show
“And in the Autoresearch experiments later in the paper, the variance across seeds is enormous.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script