Definition
Plain language
A well-known study that worked out the best way to split a training budget between making a model bigger and feeding it more data.
As stated in the literature
DeepMind's compute-optimal scaling result showing that, for a fixed training budget, model size and token count should be scaled in roughly equal proportion; invoked as the template for deriving harness and dictionary scaling laws.
Also called: Chinchilla scaling
Why it matters: It tells teams how to split a fixed budget for the best model, avoiding the costly mistake of building something too big but undertrained.
For example, it implies that doubling your training budget is best spent making the model somewhat bigger and feeding it proportionally more data, rather than just inflating model size.
Heard on the show
“They treated it like any other big machine-learning problem and derived scaling laws — in the same spirit as the famous Chinchilla work on training large models.”Episode 098 — Finding Millions of Readable Concepts Inside a Real, Deployed AI Model