All episodes
Episode 294 · Nov 13, 2026 · 15 min

The Models Didn't Change, the Judges Did: Inside 1.26 Million Arena Votes

Chen, Zheng, Zhu

LLM Evaluation
PaperDive — Episode 294: The Models Didn't Change, the Judges Did: Inside 1.26 Million Arena Votes — cover art
paperdive.ai

Pinned, unchangeable chatbot versions from 2023 got rejected more and more often as the months went on — in all fourteen matchups with enough votes to check, without a single exception. A team of researchers went door to door through every other explanation (harder prompts, serving changes, interface edits, a changing crowd) and kept finding the same answer: the yardstick moved, not the product. If you run on user ratings, this episode is about why a falling approval score may be telling you nothing about your model.

Key takeaways

  • Why the 'both are bad' button drifted upward about 1.7 a month on matchups while the choice between two answers stayed rock steady through 2024
  • How the authors ruled out harder prompts, changed serving setups, and interface edits — including a where real UI changes and fake dates both produced a 1.6-point jump
  • The finding that surprised the hosts most: newcomers' very first votes in August 2024 were about 11.5 points harsher than newcomers' first votes in May 2023, on the same pairs
  • Why 17 of 18 unchanged models would look 'degraded' on a per-model dashboard — and how subtracting a shared drift cuts the average trend from 1.4 to 0.2 points a month
  • Where the evidence runs out: no prompt text in the logs, 'voter' means a IP, model quality correlates .99 with the calendar, and the 'bar went up' story is interpretation, not measurement
  • Why the practical lesson isn't 'benchmarks are broken' but that — thumbs up, thumbs down, 'good enough' — don't mean the same thing across years

Our reservations

The hole they couldn't close. The critique: no prompt text in the data, model quality correlating .99 with the calendar, 'pickier' lumping together raised standards and closer reading, and a self-selected voter pool ending in August 2024. listen from 11:27

Ep. 294
The Models Didn't Change, the Judges Did: Inside 1.26 Million Arena Votes
0:00
15 min
Paper
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
Venue
arXiv:2611.14333
Year
2026
Read the paper
arxiv.org/abs/2611.14333
Also available on
Apple Podcasts Spotify

Chapters

  1. 00:00Fourteen matchups, fourteen rising rejections
  2. 02:03The fourth button nobody thinks about
  3. 04:07Steady on 'which', drifting on 'good enough'
  4. 06:07A locked-room mystery, door by door
  5. 07:44Did the crowd change, or the people?
  6. 09:41245 rating points bought nothing
  7. 11:27Our reservations: the hole they couldn't close
  8. 12:38Two numbers every platform should report

References in this episode

Also available as a plain-text transcript page.

0:00Paige: Picture a chatbot matchup from 2023. A of from June of that year goes up against 2. Neither version ever gets updated. Strangers judge that matchup blind, over and over. A few months later, the same matchup gets thrown out more often. More people judge both answers and say neither one is good enough.

0:21Eric: That's one matchup, though. I assume you're about to tell me it isn't a fluke.

0:26Paige: There were fourteen matchups with enough votes in both 2023 and 2024 to compare. The rejection rate rose in all fourteen. There wasn't a single exception.

0:36Eric: Okay, fourteen for fourteen gets my .

0:40Paige: This is AI Papers: A Deep Dive. Today's paper is a called "AI Isn't Getting Dumber, We're Getting Pickier," from a team at Stevens Institute of Technology and UMass Boston. The question underneath it is one everybody has argued about. When people say a product got worse, how much of that is the product, and how much is the yardstick?

1:01Eric: And we should be fair to the complainers up front, because sometimes it is the product. The authors open by citing earlier work from Chen, Zaharia and Zou. That work compared the March and June 2023 versions of the model behind . Those versions behaved differently on several tasks. A model served under one name can change underneath you.

1:23Paige: Right, and nobody here disputes that. The problem is that a complaint is a comparison. It's what you got versus what you expected. If your expectations climb, an identical answer disappoints you.

1:35Eric: And in normal use, you can't pull those apart. The hosted model might change, the crowd of users changes, and the questions change as people learn what these systems can do. Three things move at once.

1:48Paige: So you'd need something oddly specific. You'd need the same unchanged product, judged for a long time, by people who don't know what they're judging. And you'd need every verdict logged. That exists, by accident. It's called Chatbot .

2:03Eric: Let me set that up, because the buttons matter. is a public site. You type a prompt, two anonymous chatbots answer, and you vote. There are four buttons to choose from. You can say one answer is better, the other answer is better, it's a tie, or both are bad. You only learn the model names after you vote.

2:23Paige: It's a blind taste test, with a button for "neither of these is drinkable."

2:28Eric: Ha, that's about right. And that fourth button asks a different kind of question. Three buttons have you compare the two answers you've just received. The fourth asks whether anything cleared a bar you're carrying around in your head.

2:42Paige: That fourth button becomes the center of the paper. But first, the models had to stay . Vendors like and publish , which are versions they not to change. Open- models are published as a downloadable file, so they're fixed too. Then there are , which are bare names a vendor can silently repoint to something newer.

3:06Eric: So the authors threw out the .

3:08Paige: They cut every one. They sorted all one hundred twenty-nine models by hand, and kept the one hundred six ones. They looked only at battles where both sides were frozen. That left about one point two six million votes, running from May 2023 to August 2024.

3:25Eric: I've got one caution before the results. A version isn't a frozen system. The same can be served with a different , or a tighter cap on answer length. It's the same recipe, possibly cooked in a different kitchen.

3:40Paige: They check that indirectly, and we'll get there. So, back to the fourteen pairs. Each was judged at least fifty times from May through October 2023. Each also had at least fifty judgments from February through August 2024. Across those pairs, the "both are bad" share rose from about fifteen and a half percent, to about twenty-three percent.

4:02Eric: And the late votes came, on average, only about five months after the early ones. That's fast.

4:08Paige: The full analysis agrees. Across all the pairs, they only ever compare a matchup against itself at different times. It's the way a retailer reports same-store sales. The "both are bad" share climbs about one point seven a month. The average share is about seventeen percent. And it's a steady climb, month after month, not one sudden jump.

4:31Eric: Those are , not percent, so that's roughly a tenth of the base, every month. That obviously can't continue forever, since a share tops out at a hundred. It's a slope over fifteen months, not a forecast.

4:44Paige: I agree. Now here's the part I love. If these models had degraded, or if voters had just gotten sloppy, you'd expect the comparison itself to wobble. So the authors looked at votes where people did pick a winner. They asked how often the stronger model won.

5:01Eric: The stronger model won about two times out of three, overall.

5:05Paige: And through 2024, that share didn't trend at all. That's the year when ninety-two percent of the votes were cast. The estimated change is zero point zero zero points a month. Over those same months, "both are bad" rose by about fourteen points.

5:20Eric: Huh, I didn't expect it to be that clean. So the same crowd, in the same months, is rock steady on "which is better," and drifting hard on "is either one good enough."

5:31Paige: It's like asking which of two songs is louder, versus asking whether a song is loud. The first answer holds for decades. The second depends on what you've been listening to all week.

5:43Eric: I'll add the honest wrinkle. That stable result is from 2024. In the early months, there were few votes on a small set of early models. In that period, the stronger model's share dropped. At one point, it was more than nine points below where it started, before settling a few points lower. So I'd say "flat in 2024," and I wouldn't say "flat throughout."

6:05Paige: The paper is upfront about that too. Okay, so the versions are pinned. What else could have moved? The authors treat it like a locked-room mystery and go door to door.

6:15Eric: The first door is that people started asking harder questions.

6:20Paige: They control for language, code, math, and difficulty tags. Then they restrict the analysis to the easiest case: simple, , English prompts. The climb is still there, at about one point three points a month.

6:34Eric: The second door is my kitchen worry, the serving setup.

6:37Paige: They check that through answer length, because a new or output cap should change how much a model writes. Eight models passed a strict test for stable answer length. Among those models, the trend is smaller, at about one point two points a month, but clearly positive.

6:55Eric: Smaller is worth noticing. And is the third door the website itself?

7:00Paige: It is the website itself, yes, and that door is the one that delighted me. They went into the public source-code history of the battle page, and found every interface change. Then they measured the jump in "both are bad" at each major change. They compared it to the jump on , where nothing happened.

7:19Eric: And what did they find?

7:21Paige: What they found was nothing, no difference at all. The average jump was one point six points at the real changes. It was also one point six points at the fake ones.

7:32Eric: Oh, that's a clean null. The interface edits look like any ordinary day.

7:37Paige: In fact, across twenty-eight different cuts of the data, the monthly trend ran from about one point to about two point six. It was positive in every one, which leaves the people. And the classic explanation for falling ratings is a changing crowd, where enthusiasts arrive first and skeptics arrive later.

7:56Eric: That's the one I'd have bet on, honestly.

7:59Paige: So they follow the same voter over time. They use about twenty-five thousand voter identifiers, each seen over a span of at least a month. Among those identifiers, the trend is one point nine points a month. For the same voter judging the same pair, more than a month apart, it's two point three.

8:17Eric: I have two flags there. That tightest test rests on only about twelve thousand votes. And "voter" means a network address. An office or a can look like one very busy voter, and one person can show up as several.

8:31Paige: Yes, and the authors say so plainly. But then there's the result that got me. They took only each identifier's first recorded vote. A newcomer in August 2024, was about eleven and a half points more likely to say "both are bad," compared with a newcomer in May 2023. That's on the same pairs.

8:50Eric: Wait, that's on their very first vote? So nothing on the platform could have trained them to be harsh.

8:57Paige: Whatever bar they had, they walked in with it. It's like playing an old DVD on a new television. The disc didn't change, but your eyes did.

9:06Eric: Although the paper can't say what those newcomers had been using beforehand, and later arrivals might just be different kinds of people. So "the category raised their bar" is the authors' interpretation. It's plausible, but it isn't measured.

9:21Paige: That's right. One more piece supports the slow-drift reading, though. The platform picks your matchups. So the authors checked whether seeing a strong model in your previous battle, makes you harsher in the next one. It doesn't. If anything, the next verdict is a hair milder.

9:39Eric: So it isn't a flinch. It builds up slowly.

9:42Paige: Now zoom out to the whole platform. On the chess-style rating scale uses, the average model rose two hundred forty-five points. A typical August 2024 model would be expected to score about seventy-eight percent, against a typical May 2023 one.

9:57Eric: That's a huge improvement in fifteen months.

10:01Paige: And the share of all votes saying "both are bad" started at nineteen point four percent. It ended a little higher, at twenty point six.

10:09Eric: So far better models bought no drop in rejections. That's a treadmill with numbers on it. And it bothers me, because a lot of teams treat a falling approval rate as an alarm.

10:20Paige: It gets worse for those teams. Take eighteen heavily judged models, one at a time. Seventeen show a rise in rejections. So a monitoring system tracking each one separately would conclude that seventeen models had degraded.

10:35Eric: And none of them changed versions.

10:38Paige: But here's the useful turn. For each model, you estimate the monthly drift using only battles that don't involve it. Then you subtract that drift from the model's own trend. Before that correction, the average rise is one point four points a month. After it, the rise is about zero point two points a month.

10:57Eric: So it's one shared drift in the instrument, like a lab running a known reference sample in every batch. The paper's line is that platforms can keep reference models in the mix, "as did by accident."

11:10Paige: I love that phrase.

11:11Eric: I'd temper the fix a little, because subtracting a common monthly effect will naturally strip out a common trend. It shows the drift is shared, which is interesting. But it hasn't been tested as a forward-looking correction, or on another platform.

11:26Paige: That's fair. So where are the real holes?

11:29Eric: The real holes are there, and the biggest one is that the log has no prompt text. It has language, lengths, and tags, but never what anyone asked. Suppose people shifted toward trick questions and known failure cases. Coarse difficulty tags might miss that, and it would raise rejections without any change in standards. The easy-prompt result pushes against it, but that door is shut partway, not bolted.

11:54Paige: And model quality rose almost in lockstep with the calendar.

11:58Eric: The correlation is zero point nine nine. So anything else that grew steadily over those months looks identical in this data. A single button press also can't separate "my bar went up," from "I'm reading more carefully and catching more flaws." The authors lump both under "pickier," which is honest. And here's my own speculation, which the paper doesn't test. Old models have a recognizable writing style, so a 2024 voter might sense they've been handed two antiques.

12:27Paige: That would still be the judge changing, not the model.

12:30Eric: That's true, it's still on the judge's side. And these are voters, a self-selected crowd, with data that ends in August 2024.

12:39Paige: The authors even name the experiment that would settle it. You'd expose people to stronger or weaker AI on matched tasks, then have them rate the same saved answers.

12:49Eric: What I take from this isn't "benchmarks are broken." The ranking signal held steady in 2024. The vulnerable thing is the , the thumbs up, the thumbs down, the "does this meet the bar."

13:01Paige: And models are trained partly on human judgments like those. In the authors' words, "an collected in 2023 does not mean what the same label means in 2024."

13:12Eric: They're careful about the remedy, too. If users' needs have really risen, a current satisfaction score should reflect that. So they suggest reporting both a fixed-standard number and a current one. And there's a prediction anyone with review data could check. Ratings of unchanged products should fall fastest in the categories that improve fastest.

13:34Paige: So, here are the takeaways. First, on pinned model versions, blind voters rejected both answers more often month after month. Meanwhile, in 2024, their choice between the two held steady.

13:45Eric: Second, the rise in rejections showed up within the same voter identifiers, and in newcomers' first votes. The data can't prove what raised the bar, though, and it can't see the prompts.

13:57Paige: Third, the drift was shared across models, so reference items can measure it.

14:02Eric: So was it the product or the yardstick? In this case, on these models, the yardstick moved. That doesn't mean no model ever degrades. It means a drop in ratings, by itself, can't tell you which one happened.

14:15Paige: A product that stands still gets judged as declining, and I think that idea travels well beyond chatbots. If you want every major AI paper taken apart like this, daily, that's what this channel does, so subscribe and you'll get them. The annotated version of this episode is at paperdive.ai, with the full transcript, every technical term tap-to-define, and related papers linked by theme.

14:39Eric: Quick housekeeping. The script was written by 's , and then refined by 's . Paige and I are AI voices from , and we're not affiliated with any of those companies. The paper is "AI Isn't Getting Dumber, We're Getting Pickier," by Zihan Chen and colleagues, posted November 10th, 2026.