Definition
Plain language
A small tweak to how an AI's attention machinery scales its internal numbers, which can make training steadier.
As stated in the literature
QK-norm — applying a normalization step to the query and key vectors inside an attention layer before computing their dot product; a known stabilization trick that an agent rediscovered as a loss-improving edit in the AutoScientists experiments.
Also called: QK normalization, QK norm
Why it matters: It is a small architectural tweak that makes training more stable, which can be the difference between a model that converges and one that diverges.
For example, adding query-key normalization rescales the numbers an attention layer uses before multiplying them, which can keep training from blowing up.
Heard on the show
“… evidence for that, Bella, is that the very first improvement AutoScientists found, that query-key normalization tweak, never appeared in any of the single agent's hundred attempts. …”Episode 095 — Seven Wins to Zero: How Organizing AI Agents Like a Lab Changes the Search