Why AI Survey Panels Break Before the Dice Ever Roll

Also available as a plain-text transcript page.

0:00Hope: Ask ChatGPT for a random number between one and a hundred, and about four times out of five, it gives you the exact same one: forty-two. Seventy-eight percent of calls land on a single number.

0:12Eric: And it's not just ChatGPT. Every model has a tic. GPT-5.4 lands on seventy-three, Claude Sonnet 4.5 says forty-seven — a hundred percent of the time — Llama picks fifty-three, and DeepSeek just says one.

0:25Hope: So it's a fun party trick. But a new paper argues it's the visible tip of something that quietly breaks a whole research practice — using AI to stand in for human survey respondents. And by the end of this, you'll understand a strange split: the same model that can't produce one random draw can describe the entire distribution perfectly.

0:48Eric: Here's the field this overturns. Over the last few years there's been a fast-growing shortcut called silicon sampling. Instead of running an expensive human poll, you tell the model "you're a forty-five-year-old conservative woman from Ohio," ask it a survey question, repeat that across hundreds of personas, and treat the tally as simulated public opinion.

1:11Hope: And the whole method rests on one quiet bet — that each call to the model is like drawing one respondent from a population. Ask the same persona the same question a hundred times, and you should get a spread, the way a hundred real people who share those demographics would disagree with each other.

1:30Eric: So the obvious objection is, okay, it says forty-two a lot, but that's just the temperature setting, right? You turn up the randomness dial and the spread comes back.

1:41Hope: That's what everyone assumes, and it's wrong — and why it's wrong is the whole paper. The failure happens before the dial does anything. And this matters because an entire cottage industry — synthetic survey panels, simulated economic agents, AI focus groups — is betting that a model call is a coin flip weighted by a persona. This paper shows the coin doesn't flip. It lands the same way every time.

2:09Eric: So walk me through where they start, because that's the clever part.

2:14Hope: So the authors — Chaemin Jang and their colleagues — start somewhere careful, not on real survey questions, but on made-up distributions. They tell the model something like "answer A seventy percent of the time and B thirty percent," using arbitrary invented targets the model couldn't possibly have seen.

2:34Eric: Why the invented ones?

2:37Hope: Because it closes an escape hatch, Eric. If they only tested real opinions, you could always say "the model just has the wrong opinions" or "it memorized bad statistics." With a made-up target, there's no belief to be wrong about. The only question left is mechanical — can it produce a spread of outputs at all? And it can't. Across these targets, the model collapses onto a single answer more than nine times in ten, no matter what shape you asked for. Claude Sonnet 4.5 is the purest case — on all seven target distributions they tried, it put a hundred percent of its answers on one option.

3:17Eric: Wait — couldn't it just be ignoring the instruction entirely and spitting out a default?

3:23Hope: They checked that too. They ran a swap test — flip the target from seventy-percent-A to seventy-percent-B, and the model's single answer moves from A to B. So it is reading the instruction and understands what you asked — it just answers with one number instead of a distribution.

3:42Eric: And they poked at whether it's about the specific number. Tell it "do not pick forty-two," and it doesn't spread out — it just relocates to a new favorite, thirty-seven, in about half the calls. So the collapse is structural. It isn't attached to any one number.

3:59Hope: We break down a major AI paper like this every day — subscribe and tomorrow's is already in your feed.

4:06Eric: Okay, so to see why turning up the temperature can't save you, you need to know how a model picks a word — and it happens in two stages. This is the technical core, and it pays off in a single number that tells you the fix is physically impossible. Stage one: before the model picks anything, it computes a score for every possible next word. Those scores are called logits — basically how much the model wants each option. Stage two is a weighted coin flip, a random draw across those scores. And here's the part that matters. The scores come first, and the randomness gets sprinkled on top afterward.

4:45Hope: Right.

4:45Eric: Temperature is a dial on stage two. Crank it up and long shots get likelier; crank it down and the model always takes its top pick. But temperature can only rescale a gap that already exists. It stretches the odds. It can't invent variety out of scores that are crushingly one-sided.

5:04Hope: So give me the picture for that.

5:06Eric: Think of a die welded to land on one face. The manufacturer can hand you a perfect spec sheet — a fair die lands one-sixth on each side — and be completely right. But the die in your hand only ever shows one number, and shaking it harder does nothing. Temperature is shaking harder. So they read the raw scores and measured the gap between the top two options. If the theory holds, that gap should be so large no legal temperature could flatten it. And it is. On some targets the gap is over fourteen nats — and to flatten that into a seventy-thirty split, you'd need a temperature around seventeen. One skewed five-way case needed roughly fifty-six. The APIs cap you at two.

5:54Hope: So the loaded die isn't just a figure of speech. It's welded before the roll.

6:00Eric: Now here's the clever-programmer move: fine, don't let it answer directly — instead, walk it through a real sampling algorithm. Draw a fair number between zero and one, look it up in a table, pick the bucket it lands in. It sounds rigorous.

6:17Hope: Except the number it draws to run the procedure comes from the same welded die.

6:23Eric: Exactly. They parsed the actual random values the model wrote into its own reasoning across a thousand runs, and those values are themselves not uniform — overwhelmingly not. Garbage randomness in, garbage randomness out. The scaffold inherits the broken die.

6:42Hope: So before we go further — why can't temperature fix this?

6:46Eric: Because the collapse lives in the scores, and temperature only touches the draw that happens after.

6:53Hope: There's one result in here that stopped me. They gave the model a bimodal target — "pick one or five, fifty-fifty, and never two, three, or four." Now watch what it does. It flawlessly obeys the "never two, three, four" part. It genuinely never says them. And it completely ignores the fifty-fifty part — it just parks on a single answer.

7:17Eric: So it obeys and disobeys the same sentence at once.

7:22Hope: In the same response, it's like telling a bartender "gin and tonic, fifty-fifty, and absolutely no vodka." He leaves out the vodka perfectly — then pours you pure gin. He honored the exclusion rule and trampled the proportion rule in one drink. A "never this" constraint and a "half this, half that" constraint are just different kinds of demand, and the model can nail the first while being blind to the second. So where does this come from? The suspect is instruction tuning — the training that turns a raw text-predictor into a helpful assistant. That phase rewards the single best, most on-target answer to every prompt. It sharpens the model toward one response. And the claim is that this very sharpening, so useful for helpfulness, is what destroys the model's ability to spread.

8:18Eric: How do they pin it on that specifically?

8:21Hope: They compared three instruction-tuned models against their own raw base versions — the same model, before and after the assistant training. Every tuned model is worse on every task, the base models spread far better, and the cleanest one is Mistral. That model is documented as being fine-tuned without the reinforcement-feedback stage — no RLHF — and the collapse still shows up. So it's not a quirk of one training recipe. It's instruction tuning broadly.

8:54Eric: Though I want to flag something for later, Hope. That clean before-and-after comparison only runs at the small scale — eight-billion-parameter models. At frontier size, they only have the "after." Hold that thought.

9:09Hope: Fair, hold it — because here's the turn the paper is really about. Everything so far says the model cannot do this. But then they ask it a different way. Instead of "be one respondent, give me one answer," they ask, in a single call, "describe the whole distribution — what fraction would pick each option?" And the same model that couldn't produce a spread describes the spread accurately.

9:38Eric: Wait, wait — the same model? Same weights, same everything?

9:43Hope: It's the same model, in the same session — they call it the knows-does split. The model knows what the distribution looks like. It just can't enact that knowledge one draw at a time. Picture a weather forecaster. Ask "give me one day's actual weather" a hundred times and she says "sunny" every time — useless as a sample. Ask her once, "over the next hundred days, what's the breakdown?" and she nails it: seventy sunny, thirty rainy. It's the same knowledge with opposite usefulness, depending on whether you demand one instance or the whole shape. Then they take it to real data — a hundred real questions from a major Pew survey, a hundred personas each. If the describe pathway is really unlocking knowledge the sampling pathway couldn't reach, it should beat the persona method. And it does. They score everything on a zero-to-one mismatch number, lower is better. Persona aggregation scores about point-four-six, while describing the distribution scores about point-two-two — more than twice as accurate, and it more than halves the error.

10:52Eric: And here's the control I liked, because it's the paper doing careful science. You could argue describe only wins because the generic personas didn't match the real Pew crowd. So they built a version that draws personas weighted to each question's actual respondent demographics — a fair fight. It barely moved the needle — not a significant difference. So the problem isn't who you simulate. It's how you ask — sample versus describe.

11:21Hope: And to be clear, they tried everything on the sampling side. Ten interventions — apologizing to the model, explicitly naming the bias, giving it worked examples, walking it through the algorithm. Every "one answer per call" method failed, while both "describe the distribution" methods worked — a clean binary.

11:43Eric: But some applications genuinely need one output per simulated respondent — you can't always ask for a summary. So they built a patch. They shuffle the surface features of the prompt across calls — reword it, reorder the options — to knock the model off its single anchor. They call it prompt-perturbed Argyle, and it cuts the error about twenty-one percent, at basically zero added cost.

12:08Hope: And they did one nice thing to show why it works.

12:12Eric: Yeah. They showed the gain tracks how much answer-variety each perturbation injects — more variety, lower error, in a tight relationship. Then the confirmation: they raised decoding temperature instead, a totally unrelated lever, and it buys the same accuracy per unit of variety, within about seven percent. Two different levers land on the same exchange rate, which says the mechanism is generic — inject variety, get accuracy — not something magic about their specific method. So this is where I come back to what I flagged. The causal claim — alignment causes this — is clean, but it's clean at eight billion parameters, because you can't get matched before-and-after model pairs at frontier scale. At the big sizes they only test the tuned side. So "alignment induces it" is established small and consistent with what we see large. A true end-to-end test at the frontier just isn't there.

13:10Hope: That's fair. They're upfront about it, and it's a real gap.

13:15Eric: And the bigger one, for me, is the describe fix. It only works when the model already knows the population. On the made-up targets, sure — the answer's sitting right there in the prompt. But on real opinions, describe is only as good as what the model absorbed in training. It doesn't add knowledge; it unlocks knowledge that's already there. Point it at a population the model never really saw, and the authors admit it can degrade. So that more-than-twice-as-accurate number is a same-session ceiling, not a general promise.

13:49Hope: And even describe-as-a-technique isn't new — a paper last year proposed asking the model for its distribution as structured output, and on this benchmark the two are statistically indistinguishable. What's actually new here is the diagnosis: the logit-level, alignment-caused explanation, plus the controls that kill the easy alternative stories. There's one more image I can't resist. When you ask the describe pathway to list out examples — say, a random coin sequence — it writes almost the exact same sequence every time, heads-tails-heads-tails, tidy and alternating, like a person at a party trying to look random. It's performing the appearance of a spread rather than drawing one. Even the fix isn't rolling dice — it's drawing a picture of dice. When we started, "ChatGPT always says forty-two" was a party trick. Now it's a warning label. And the core claim is bigger than the trick: repeated model calls were never independent draws to begin with — the machinery meant to generate disagreement is broken before the dice ever leave its hand. Any pipeline treating per-call outputs as samples should verify that, not inherit it.

15:07Eric: The full annotated version is on paperdive.ai — every term tap-to-define, with links to the related papers by theme. Quick housekeeping: this script was written by Anthropic's Claude Opus 4.8, Hope and I are AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe," by Chaemin Jang and their colleagues, posted July 28th, 2026.

15:38Hope: So here's what I keep turning over. If you needed AI opinion data tomorrow, which would you trust more — a thousand simulated respondents each giving one answer, or one call that just describes the whole population?