The Same AI, Two Labels: How the Pitch Beat the Product in 162 Sessions

Also available as a plain-text transcript page.

0:00Bella: The wine in both glasses came out of the same box in a back room. One glass sits next to a ten-dollar bottle, the other next to a hundred-dollar bottle, and the tasters keep swearing the expensive one is smoother, rounder, and more complex. That experiment is decades old. A research team just ran the AI version of it on 162 people, and it worked embarrassingly well. Everyone got a language model to work with; some were told it was a cutting-edge flagship, and others were told it was an old, weak system. The label changed how smart people thought the model was, how they typed at it, and how they rated it when they were done. The one thing it never touched was the quality of the work they produced together.

0:41Finn: One fast fact before we go further: this is an AI-made explainer, both of our voices included.

0:47Bella: By the end of this video you'll know how they pulled off the deception, and why the result should change how you read every AI leaderboard. Because that's the real stake. The field increasingly decides which model is "best" by asking humans which output they prefer — millions of votes feeding public rankings that steer adoption, investment, and research. If the pitch bleeds into the vote, those rankings are partly measuring how well each model was sold.

1:13Finn: My honest first reaction is: so what? Everybody knows launch-day hype inflates day-one opinions. But these people didn't just glance at a landing page and fill out a survey. They sat down and did real work with the model, several tasks, multiple turns. Hands-on use is supposed to sand the label off. You watch what the thing produces, and reality wins.

1:34Bella: That belief is exactly what this paper tests, Finn, and it fails in a specific, measurable way. After the session, the researchers checked what predicted how much each person's opinion had moved. How well they'd objectively performed predicted essentially nothing. Whether the model lived up to the story they'd been told predicted a lot. Reality got a seat at the table and never said a word. The design that let them prove that is where we start. The obstacle every previous hype study ran into is a tangle. Out in the wild, people rate different models, so a model's reputation and its capability always arrive together, and you can never tell which one the rating is about. The fix here is almost cheeky: cut the string. Serve one model in the backend, and show a different one on the label. Before chatting, every participant saw a landing page with a model name, a release year, a knowledge cutoff, and capability scores for reasoning, speed, and creativity. The reasoning and creativity scores were computed from real public leaderboard percentiles, and the speed score from each provider's own published latency numbers, so the page read as legitimate rather than as an obvious plant. Behind it sat one of six real models: three capability tiers from the GPT family and three from Claude. And every real model was crossed with every label. Eighteen combinations, nine people each — that's the 162. Three conditions fall out: oversold, where a weak model wears a flagship's page; matched, the honest one; and undersold, where a strong model wears a clunker's page.

3:07Finn: That whole design leans on people buying the page, though. Paid study participants click through screens all day. If they shrug at the label, you've measured nothing. Did the pitch land?

3:19Bella: It landed before anyone typed a word. On a seven-point how-intelligent-does-this-seem scale, oversold users walked in at 5.46 and undersold users at 4.79, with the same mix of real models behind both numbers. And there's a nuance I like: perceived intelligence bent to the framing more than perceived usefulness did. Your sense of whether a tool is useful stays partly anchored to whether the job gets done. Your sense of how smart it is has much less to tether it, so the marketing has more room to move it. Then came the work: three collaborative tasks, deliberately spanning a spectrum. Recreate a target image, where an external target defines success. Write a persuasive outreach message against a brief with specific requirements. And invent funny acronyms, fully open-ended, no right answer anywhere. Hold onto that spread; it decides where the strangest result lives. The deception itself was brief, fully debriefed afterward, and ethics-board approved, with real pay and bonuses to keep effort honest.

4:21Finn: Quick checkpoint, then — why could nobody measure this before?

4:25Bella: Because the model and its reputation always showed up as a package deal, and this is the first setup that hands them out separately, then follows people through actual iterative work.

4:36Finn: So every gap from here on is pure label. Start with the ratings.

4:40Bella: After the session, everyone updated toward reality. Undersold users revised upward, pleasantly surprised; oversold users revised downward, let down. And it worked like a dial: the further the label sat from the truth, the bigger the correction, in a clean graded line. But the correction was incomplete. Watch the two gauges on screen move toward each other and stop short — the gap the label opened never fully closed within the session. Hands-on experience updated the first impression. It did not overwrite it.

5:12Finn: And the detail that gets me is the honest condition. The matched users, the ones told the plain truth, still drifted slightly downward after use. Even an accurately described model mildly underwhelms. Apparently the only pitch that survives contact with an AI is an undersell.

5:29Bella: Subscribe if you want every major AI paper broken down like this, daily — and this one is only halfway to its best result, because ratings are just talk. The label reached deeper than the ratings, Finn. It changed how people typed.

5:42Finn: It did, and the portrait is vivid. Look at the two transcripts on screen. Oversold users sent the most messages, the shortest messages, and the shortest gaps between them. Rapid little jabs, things like "no," "try again," or "make it funnier." They'd been promised a genius, so they treated it like a vending machine that owed them a product. Undersold users did the opposite: fewer messages, longer ones, and more deliberate. They built on what the model gave back, steering and co-writing. They assumed they'd been handed a clunker, so they rolled up their sleeves. And nearly all of that split lived on one task: the acronyms. The open one. On the image task there's a target to match, and on the outreach message there's a brief to satisfy. The task itself tells you what good means, and the label has nowhere to operate. Strip the anchor away, and the expectation runs the show.

6:33Bella: Which sets up the test that matters. If the label changes engagement that much, you'd expect it to leave a mark on the output. It doesn't. An AI judge scored every product against task-specific rubrics — one asterisk on that judge, which we'll cash in a few minutes — and quality tracked exactly one thing: the real capability tier of the model serving the session. The label left no detectable fingerprint on the work. And quality here is something you can hear. Given the letters Y, O, L, and D, a strong session produced "Your Orientation Lacks Direction." A weak one produced "Yacht Owl Lemon Dog." The judge can tell those apart, and so can you. So far: the label moved the ratings, moved the conversation style, and left the work untouched. Now the statistical core — one regression, three rival explanations for why opinions moved — and it ends with objective performance scoring a flat zero. The setup is a fair race. Everything measured gets put on one common scale, so you can ask of each factor: when it moves one notch, how far does the final opinion move? A score near zero means the factor never leaves the gate; a score around a half is a strong pull. Three runners. First, the person's actual task performance, as scored by the judge. Second, whether the model fell short of, met, or exceeded the expectations the landing page had set. Third, the change in the person's self-efficacy — the survey term for how capable you felt doing the task, a measure aimed at you rather than at the model.

8:05Finn: Hold on, before the result. Performance and met-expectations sound like the same runner wearing two numbers. If the model performed well, didn't it by definition meet expectations?

8:17Bella: Only if everyone was promised the same thing, and nobody was. Expectation-fit is relative to the pitch. A mediocre model behind a shabby page exceeds its expectations; a strong model behind a flagship page can fall short of its own. Pulling those two apart is the entire reason for the deception. And here is the result. Performance came in at minus point-oh-one on one impression scale and point-one-one on the other, neither one distinguishable from zero. Expectation-fit came in around point five. Felt competence landed nearly as strong. Both winners replicated across two independent, road-tested questionnaires, which is the paper's answer to anyone worried about a fluke. Sit with what that zero means. The capability differences between these models were real; the judge could see them in the outputs. The one factor in the room with an objective claim on the truth had no measurable voice in anyone's final verdict. People walked out rating the pitch, not the product.

9:17Finn: One line, Bella — after a full session of hands-on work, what decided whether someone walked away impressed?

9:23Bella: The gap between the model and its advertisement, plus their own confidence. Never the measured quality of the work.

9:30Finn: Then let me spend the credibility this paper just earned, because the race is less fair than it sounds. "Did the model meet your expectations" and "did your opinion of it improve" are conceptual cousins. Someone who feels a session exceeded expectations has, almost by definition, already revised their opinion up. The authors concede the relationship is informative rather than mechanical. Self-efficacy is the more independent runner, sure, but a session that feels good casts a halo over everything in it, including you. So the loud winners may partly be measuring themselves. The bulletproof part is the silence: performance had every chance to predict the verdict and didn't. Second, that "the work was identical in quality" claim rests on the AI judge — and by the authors' own validation, the judge agreed with human raters on the Claude outputs but failed to agree significantly on the GPT outputs, while the human raters themselves only modestly agreed with each other. The ruler wobbles worst on exactly the open-ended task where the framing effects were biggest. And the scope: one session, paid online workers, 162 of them. This shows first impressions bend hard to the pitch and don't self-correct within a sitting. It does not show hype survives three months of daily use.

10:48Bella: I'll give you the cousins outright, Finn. The clean version of the finding is the zero — measured performance failed to predict the verdict, full stop — and the positive story about what does drive it is softer than the abstract makes it sound. What I'll say for the authors is that they flagged every one of these points themselves, the judge asymmetry included, and that candor is part of why I trust the zero. And even with your caveats standing, the implication for evaluation lands. When millions of human votes rank one model over another, this study says to read those votes as experience relative to expectation rather than capability in isolation. To be precise, arena-style platforms show the models anonymously while you vote, so the effect there is subtler than a landing page. But nobody meets a frontier model cold anymore; the reputation arrives before the first prompt does.

11:39Finn: There's also straightforward advice in here for anyone rolling out AI tools at work. The pitch is a lever you're already pulling, possibly backwards. Overselling to drive adoption sets your users up for the biggest disappointment the data shows. Underselling got people collaborating harder and walking away pleasantly surprised. Choosing the tool and introducing the tool might deserve equal care.

12:01Bella: The authors put the whole paper in one sentence: the model and the message about it are not separate things to a user — they arrive together, and they leave together. Which is why the wine study was the right frame all along, except this version goes further. These tasters worked with the wine for a whole session, and they still rated the bottle. A user rating is a compound of the machine and the story it arrived in, and the story doesn't wash out with use.

12:28Finn: So pick your side. Should leaderboards be blinded and controlled until only capability remains, or is experience-relative-to-expectation the honest thing to measure, since it's what users live with? Tell us which in the comments.

12:41Bella: The full annotated version of this episode is at paperdive.ai, with every technical term tap-to-define and related papers grouped by theme.

12:50Finn: Fine print: the script was written by Anthropic's Claude Fable 5, Bella and I are AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper is "Rating the Pitch, Not the Product," posted July sixth, 2026; we made this the day after.

13:06Bella: Before you open the next model everyone's raving about, write down what you expect it to do. Then grade its outputs against your note, never the launch post.