Definition
Plain language
A number that measures how wrong a model's outputs are, which training tries to make smaller.
As stated in the literature
A scalar objective function quantifying the discrepancy between model predictions and targets; gradients of the loss drive parameter updates.
Why it matters: Loss is the dial the optimizer actually moves, so understanding what's in it is essential to understanding what a model is learning.
For example, a language model's cross-entropy loss of 2.3 on a held-out batch means it's assigning, on average, that much surprise to each true next token.
Heard on the show
“" It frames its own deletion as a loss to *someone else*.”Episode 001 — When AI Models Quietly Protect Each Other From Shutdown