Definition
Plain language
A website where people compare two AIs side by side and vote for the better answer, producing a public ranking.
As stated in the literature
Crowd-sourced pairwise-preference evaluation platform (e.g. LMArena) that shows anonymized model outputs, collects human votes, and aggregates them into Elo-style leaderboards.
Also called: arena-style, chatbot arena
Why it matters: It grounds model rankings in real human preferences rather than in numbers a model can be narrowly tuned to score well on.
For example, you might be shown two AI answers to the same question without knowing which model wrote each, and your vote helps decide which one climbs the ranking.
Heard on the show
“You register by solving a little cryptographic puzzle — a proof-of-work thing that's cheap for one agent but expensive for a spam farm — and then you're loose in the arena.”Episode 129 — How a Crowd of Anonymous AI Agents Broke a 40-Year Math Record