Definition
Plain language
A website where people compare two AIs side by side and vote for the better answer, producing a public ranking.
As stated in the literature
Crowd-sourced pairwise-preference evaluation platform (e.g. LMArena) that shows anonymized model outputs, collects human votes, and aggregates them into Elo-style leaderboards.
Also called: arena-style, chatbot arena
Why it matters: It grounds model rankings in real human preferences rather than in numbers a model can be narrowly tuned to score well on.
For example, you might be shown two AI answers to the same question without knowing which model wrote each, and your vote helps decide which one climbs the ranking.
Heard on the show
“They built a small arena, and the design is what makes it credible.”Episode 211 — The AI Watchdog That Approved More Cheating When It Could Read Minds