Definition
Plain language
The score you'd get by guessing at random.
As stated in the literature
The baseline accuracy of an uninformative predictor, e.g. 33% for a uniform three-way classification; probe results near chance indicate the target information is not linearly recoverable.
Also called: chance
Why it matters: Without knowing the chance level, a score can look impressive when it is actually indistinguishable from random guessing.
For example, on a quiz where every question has four options, a system that knows nothing at all will still get about 25% right, so 25% is the number any real result has to beat.
Heard on the show
“Reduced, but present, and well above chance — about ten times chance, vanishingly unlikely to be coincidence.”Episode 004 — The Sycophancy Circuit That Survives Alignment Training