Definition
Plain language
A way to compare neural architectures by holding the compute budget fixed and seeing which scales best.
As stated in the literature
An analysis methodology training models at multiple sizes under matched compute budgets, computing the compute-optimal frontier, and comparing architectures by their loss-versus-compute slope.
Why it matters: It's the standard fair comparison for architectures, since otherwise a model might just be winning because it was trained with more compute.
For example, you train several sizes of both architecture A and architecture B at 10^20 FLOPs each and compare which family's best loss is lower.
Heard on the show
“The authors run what they call isoFLOP experiments: they train each architecture at three sizes — 350 million, one billion, three billion parameters — under five different compute budgets.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script