Definition
Plain language
Picking the best of many AI answers by reading all their full reasoning side by side and reasoning about which argument actually holds up, instead of just counting votes.
As stated in the literature
A selection method that concatenates every candidate's full reasoning trace into one judge prompt and asks the judge to jointly compare, rank, and even synthesize a new answer; contrasted with majority voting, which buries a lone correct minority answer, and used to beat frontier single models on ARC-AGI-2.
Why it matters: It rescues correct-but-rare answers that simple vote-counting would drown out, letting a group of models reach conclusions no single one could.
For example, instead of counting how many AI answers say '42', a judge reads every answer's full reasoning and picks the one whose argument actually holds up, even if only one answer got it right.
Heard on the show
“Diverse generation, then holistic judging.”Episode 191 — How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them