Glossary · Term

error bars

← all terms

Definition

Plain language

The little range around a reported number showing how much it might wobble if you measured again.

As stated in the literature

A visual or numeric representation of estimate uncertainty (standard deviation, standard error, or confidence interval); single-seed results carry none, so differences of a few points may be path dependence rather than signal.

Also called: error bar

Why it matters: Without them, small gaps between systems look like real differences when they may just be luck, which quietly corrupts every leaderboard built on them.

For example, reporting a score as "72, give or take 5" tells you that a rival scoring 74 might not actually be better.

Heard on the show

“The Arena has no error bars at all.”
Episode 245 — Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It

Mentioned in 10 episodes

  1. 245
    Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It
  2. 201
    One in Four NeurIPS Papers Cites a Reference That Doesn't Exist
  3. 173
    The Free Step-Level Grader Hiding in Every RL Training Run
  4. 166
    A Router That Beats the Frontier Models It Calls
  5. 165
    A Free-Lunch Tweak That Lets a Tiny Agent Beat Frontier Giants
  6. 151
    Why More Experience Made This AI Agent Worse, And How to Fix It
  7. 149
    When Cornering a Chatbot Makes It Lie: J.P. Morgan's Case for 'Playing Dead'
  8. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  9. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
  10. 100
    How a Prompt Wrapper Lets a Frontier Model Play Poker Like an Expert

Related terms