Definition
Plain language
How well you'd do by guessing at random.
As stated in the literature
The expected score of an uninformed policy on a balanced task; fifty percent accuracy for a balanced binary label set, 0.5 AUROC for discrimination.
Also called: chance level
Why it matters: Without knowing the guessing baseline, a result like "55% accurate" can be mistaken for skill when it is barely distinguishable from random noise.
For example, on a set of yes-or-no questions with equal numbers of each, a coin flip will get about half right, so 50% accuracy tells you nothing was learned.
Heard on the show
“The test sets are balanced, so chance performance is exactly fifty percent.”Episode 251 — When a Fake Dashboard Makes an AI Agent Just as Confident