Glossary · Term

p-value

← all terms

Definition

Plain language

A number that says how surprising a result would be if nothing real were going on — small means the result probably isn't just luck.

As stated in the literature

The probability of observing a test statistic at least as extreme as the one measured under the null hypothesis; small values are taken as evidence against the null, with conventional thresholds like 0.05.

Also called: p value

Why it matters: It gives researchers a standard way to judge whether a result is likely real or could easily be a fluke of chance.

For example, if flipping a coin 20 times gives 18 heads, the p-value tells you how unlikely that lopsided result would be from a perfectly fair coin.

Heard on the show

“The correlation between tokens burned and final score is minus zero point one nine, with a p-value of zero point five.”
Episode 245 — Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It

Mentioned in 6 episodes

  1. 245
    Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It
  2. 229
    One Word Flips a Chatbot From Backbone to Yes-Man
  3. 196
    AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review
  4. 176
    An AI Designed Its Own Psychology Studies, Then Confirmed What It Found
  5. 025
    The Missing Gradient Term That Predicts Sycophancy in RLHF
  6. 015
    The Audit Number Isn't What You Think: Sycophancy and the Case Against Single-Prompt Bias Tests