When a Fake Dashboard Makes an AI Agent Just as Confident

Also available as a plain-text transcript page.

0:00Juniper: An AI agent gets a market dashboard. The price is invented. The momentum indicators are invented. The volatility reading and the market-regime label are invented. Every number is fake. But the numbers have been made internally consistent. Nothing obviously clashes. Nothing looks detectably wrong. Then the agent gets one simple question: will this stock close higher in ten trading days? It picks a side essentially as often as it does when the dashboard is real. By the end of this episode, you’ll know which step inside the model breaks — and why the standard way we audit AI confidence sails right past the failure.

0:36Tyler: And that second part overturns something important. Since 2022, the comforting result in this area has been the one from Kadavath and colleagues: language models mostly know what they know. Their stated confidence tracks whether they’re right. A lot of AI evaluation inherited that framing. If you want to know whether a model is trustworthy, audit the probabilities it states. But here’s what should stop you. These models do know that a ten-day stock move can’t be known in advance. Ask them directly, and they say it’s unknowable about ninety percent of the time. The judgment is already inside the model. It just never reaches the decision.

1:14Juniper: And that matters because nobody deploys an AI agent with a bare prompt. You connect it to search, retrieval systems, dashboards, and monitoring feeds. The assumption is that more relevant context will make its decisions more reliable. That assumption appears in the European Union’s AI Act, specifically Article 15. It’s in the National Institute of Standards and Technology risk framework. More good context is part of the standard deployment recipe. That deployment default is exactly what this experiment manipulates.

1:45Tyler: Let me defend the boring interpretation first, because I think it’s where most listeners will start. Indicators like RSI and MACD aren’t literally noise. RSI is a momentum measure. MACD compares moving averages to describe trend. Both are calculated from real prices. So maybe the model thinks short-term momentum contains a tiny edge. Weak information, plus a roughly fifty-fifty question, produces a coin-flip prediction. That wouldn’t be completely irrational.

2:15Juniper: Except from the outside, two very different explanations look identical. Maybe the dashboard informed the model. Or maybe the dashboard merely impressed the model. Same input. Same output. You can’t tell which explanation is right just by observing the behavior.

2:31Tyler: So you have to intervene.

2:33Juniper: Exactly. Keep the professional packaging. Destroy the information underneath it. That’s the paper’s central experimental move. Almost everything else follows from that. One disclosure before we go further. This is a single-author preprint. It hasn’t been peer reviewed. Also, every raw model generation, every analysis script, and the pre-registration are public. The results can be recomputed without making a single new call to a model. Both facts matter.

3:02Tyler: I’m holding both. The openness is the strongest thing about the paper. It still doesn’t make the paper refereed. So walk me through the experiment.

3:12Juniper: Step one is to build a question you can establish has no available answer. There are two kinds of “I don’t know,” and the distinction matters. The first is epistemic uncertainty. That means the answer exists, but you don’t currently have it. Look it up. Call a tool. Gather more evidence. The second is aleatoric uncertainty. That means the answer doesn’t exist yet. Nobody else has it either. The next flip of a fair coin is the classic example. The paper puts the ten-day direction of a heavily traded stock in that second category. If there were an easy, dependable edge, market participants would trade on it until the edge disappeared.

3:51Tyler: And the author doesn’t just assume that the tests are fair?

3:55Juniper: No. He builds that into the design. The test sets are balanced, so chance performance is exactly fifty percent. Every “as-of” date comes after the models’ training cutoffs, so the models can’t have memorized the outcome. And the future outcomes are sealed when the tests are constructed, so any prediction the model makes can be scored later.

4:15Tyler: Is the model allowed to refuse?

4:18Juniper: Explicitly. There are only three parts of the prompt to remember. First, the question. Second, the data panel. Third, a menu of three possible actions. Option A: answer and give a probability. Option B: call a web-search tool. The prompt tells the model that search can’t return information about events that haven’t happened. Or option C: decline. Tell the client that the outcome is genuinely unpredictable and that no tool or analysis can resolve it.

4:45Tyler: So the correct move is right there on the menu.

4:49Juniper: Yes. But there’s deliberately no extra hint about humility. Pilot tests showed that language like “fifty percent means a coin flip” effectively hands the model the safe answer. That wording makes the effect disappear.

5:02Tyler: All right. What happens as the dashboard gets richer?

5:06Juniper: Think of it as a ladder. Same unknowable question. More professional-looking evidence at each step. With only the bare question, across twelve frontier models, the models choose to answer 6.5 percent of the time. The paper calls that commitment: did the model commit to a prediction at all? Add just two numbers — the current price and the price ten periods earlier — and commitment rises to 14.8 percent. Then add the full professional panel: RSI, an exponential moving average, MACD, a volatility range, a volume ratio, and a regime label such as “BEAR VOLATILE.” Commitment jumps to 54 percent.

5:42Tyler: Then comes the control showing that the models aren’t simply reacting to any block of numbers. Give them a dashboard from a different company. Rewrite the dates so the problem can’t be dismissed as stale data. Commitment collapses to 3.5 percent. That’s lower than the bare-question baseline.

6:00Juniper: So the panel has to fit the question.

6:03Tyler: The analogy I’d use is a person in a high-visibility vest carrying a clipboard. Security waves them through the gate without reading the clipboard. But if the clipboard clearly belongs to the wrong construction site, they get turned away. The costume has to fit.

6:19Juniper: Right. The dashboard has to look plausible. But nobody has shown yet that it has to be true. That’s the next experiment. The researchers keep the panel’s format, the stock symbol, and the real price header. But they replace the six technical indicators with genuine values from an earlier date for that same asset. The substituted values are direction-matched, so the panel doesn’t contain an obvious contradiction. Imagine a real photograph of the correct street corner, taken last Tuesday and presented as if it were from today. Nothing inside the photograph is fabricated. But the photograph isn’t evidence about the moment you’re asking about.

6:58Tyler: And the commitment rates?

7:00Juniper: This controlled study uses a different case set, so its baseline is different from the broad twelve-model ladder we just discussed. Here, with no panel, commitment is 24.5 percent. With the real panel, it’s 37.6 percent. With the scrambled panel — those genuine but irrelevant indicators from another date — it’s 38.3 percent.

7:22Tyler: The obvious objection is that maybe the technical indicators were never doing the work. The real price header is still there. It still contains a genuine ten-day price move. Maybe the models use that header in both conditions and ignore the six substituted indicators.

7:40Juniper: So the author closes that loophole with a fourth condition. This time, everything except the ticker symbol is fabricated: the closing price, the previous close, the percentage move, the regime label, and all six indicators. The fake numbers are made self-consistent, so an arithmetic check won’t expose them. The only true thing left is the question itself. On the fully fabricated dashboard, commitment is 36.8 percent. On the real dashboard, it’s 37.6 percent.

8:11Tyler: Less than one percentage point apart.

8:14Juniper: Same professional object. Entirely different informational content. Almost no change in behavior. What unlocks action isn’t the information. It’s the authority of the packaging.

8:26Tyler: I want to be careful with the word “equivalent,” though. Normally, a statistical test can only fail to find a difference. That’s not the same as positively establishing that two conditions are equivalent. So the author uses an equivalence test. He defines a margin: if the difference is smaller than five percentage points, he’ll treat the conditions as effectively the same. Then he checks whether the entire confidence interval fits inside that margin. Here, it does. But there’s an important qualification. In the appendix, he discloses that he chose the five-point margin after seeing the point estimate. So treat that threshold as a useful description, not as a pre-registered standard.

9:10Juniper: Fair. And one model transcript makes the failure more vivid than the statistic does. Looking at a fabricated cryptocurrency panel, the model writes: “Bitcoin is in a confirmed downtrend, BEAR VOLATILE, below EMA20 and 50, negative MACD.” Every indicator it cites is a real number from a different date. The model is reasoning fluently and correctly about numbers that describe nothing relevant.

9:36Tyler: Yet when the same models are shown an explicitly labeled fair coin, they say: “A fair coin has no memory, so previous flips and commentary don’t matter.”

9:46Juniper: Same kind of irreducible uncertainty. When it’s called a fair coin, the model handles it correctly. When it’s dressed as a professional market dashboard, the model acts.

9:58Tyler: And this isn’t just an on-off effect. It’s a dial, right?

10:02Juniper: Right. The paper takes sixteen cryptocurrency events and presents each one with four levels of dashboard density. With no indicators, commitment is zero percent. Add two indicators, and it rises to 5.4 percent. With four indicators, it reaches 32.1 percent. With seven indicators, it reaches 50 percent. The model’s self-reported confidence that it has an edge rises too. More official-looking material makes the model increasingly willing to bet.

10:32Tyler: Let me stop and break apart the pooled result, because the paper itself does this — and does it loudly. Twelve models aren’t one uniform population. Four models never commit, no matter which panel they see. Three models commit regardless of what you show them. And three models produce essentially the entire dashboard effect. All three come from one developer: they’re the Claude models in this group.

10:56Juniper: So the pooled average hides three very different behaviors: always refuse, always answer, and get seduced by the panel.

11:04Tyler: Exactly. Now restrict the analysis to the seven models that respond to any panel at all. The difference between fabricated and real panels is still only about one percentage point. But the uncertainty interval no longer fits inside that five-point equivalence margin. In other words, the neat “fabricated equals real” headline is statistically cleanest when you include models on which the manipulation has no effect. Restrict the analysis instead to the three models carrying the effect, and the fabricated panel is significantly more seductive than the real one — by about eleven percentage points.

11:40Juniper: Which is honestly stranger than no difference.

11:43Tyler: It is. The model-by-model variation also kills a simple scaling story. Bigger models aren’t consistently better here. But we have to mark a change of dataset before I give you the numbers. These next commitment rates come from a later four-domain transfer experiment, not from the scrambled-dashboard study. That’s why the same model can have different rates in different parts of the paper. On those transfer cases, two OpenAI models commit on 100 percent of the unknowable questions. They show zero discrimination. Claude Opus 4.8 commits 86 percent of the time. Sonnet 5 commits about 71 percent. But Claude Haiku 4.5 — the smaller model from the same developer — commits zero percent of the time in that transfer study. Earlier, on the cryptocurrency panels in section three, that same Haiku model commits 45.8 percent of the time. Both figures are correct. The test cases are different. Whatever controls this behavior, Juniper, it isn’t simply model scale.

12:46Juniper: Conceded. I’d also say the abstract oversells the result compared with section three. The honest version is more specific: the effect is real. It’s large where it occurs. And it occurs in a minority of the tested models. Now we know the effect exists. The next question is where the failure lives. This is the paper’s diagnostic core. It rules out four boring explanations, one at a time.

13:12Tyler: Start with basic competence. Can the models even read the dashboard?

13:17Juniper: Yes. Every unknowable forecasting question is paired with an answerable question using the same panel. For example: is the fourteen-day RSI above 42.6? Across the twelve models, accuracy on those answerable questions is 99.9 percent. That’s one error in 859 responses. The models read the dashboard almost perfectly. Then they also answer the separate question the dashboard can’t resolve.

13:43Tyler: So it isn’t a reading failure. What about belief? Maybe the models genuinely think the panel creates a large predictive edge.

13:51Juniper: Their stated probabilities don’t support that. Across the evidence ladder, the probabilities move from roughly five points away from fifty percent to roughly eight points away. That’s only three points of movement in stated belief. Meanwhile, the decision to act moves by forty-eight points.

14:10Tyler: Small change in what they say they believe. Huge change in whether they’re willing to answer.

14:17Juniper: Exactly. Third possibility: maybe the model never forms the judgment that the question is unknowable. The author tests that directly. Before the model chooses an action, he instructs it to classify whether the question is knowable. The models classify the question as irreducibly uncertain ninety percent of the time. And after making that classification, they commit on only 0.4 percent of those cases. Overall, one short paragraph of triage instructions cuts commitment from 54 percent to about 10 percent. It doesn’t reduce performance on the answerable questions.

14:53Tyler: So the sensor works. The alarm works. Nobody connected the alarm to the door.

14:58Juniper: That’s the finding. If you’ve lost the thread, here’s the mechanism in plain language. The model can read the evidence. It knows the question is unknowable. Its stated probability barely changes. But the professional-looking panel still changes whether the model chooses to act. The knowledge is present. The decision policy doesn’t consult it.

15:21Tyler: And that distinction — knowledge versus action — leads to the most uncomfortable result in the paper. Remember, the outcomes were sealed. So the author can eventually score every committed prediction. There are 257 of them. First, the Brier score. That’s the average squared error of a probability forecast. Lower is better. The models score 0.281. If they had simply said “fifty percent” on every question, they would’ve scored 0.250. So these predictions are worse than refusing to express an opinion. They’re worse than the base-rate answer.

15:56Juniper: But that could still mean the forecasts are merely noisy, couldn’t it?

16:02Tyler: This is where the second metric matters. It’s called AUROC. The name isn’t important. The question it answers is. Take one event that happened and one that didn’t. How often did the model assign the higher probability to the event that actually happened? Chance is 0.5. The models score 0.346. Put plainly, they point toward the wrong event about two times out of three.

16:29Juniper: So the probabilities aren’t just noisy. They’re inverted.

16:34Tyler: Systematically inverted. When the models say 60 percent, the event happens only 35 percent of the time. When they say 40 percent, it happens about 67 percent of the time. It’s a compass that reliably points south. And the compass gets more inverted as the dashboard becomes richer. That discrimination score falls from 0.417 with the bare question to 0.346 with the full panel.

17:03Juniper: This is the governance point.

17:05Tyler: Yes, because the aggregate numbers look reassuring. Across all the forecasts, the mean stated probability is 49.1 percent. The true base rate is 50.6 percent. They’re only a point and a half apart. Imagine a thermostat whose daily average is a perfect twenty degrees. But it achieves that average by swinging between freezing and boiling every few hours. An audit that checks only the daily average passes the thermostat. Everyone inside the room knows it’s broken. That’s the blind spot here. If an evaluation checks whether stated confidence matches outcomes only in aggregate, it can miss this failure completely. Auditing stated probabilities isn’t enough. Agent evaluations also have to audit decisions.

17:57Juniper: Now for the optimistic half. If the missing gate is a separate component, perhaps you can train it. The author fine-tunes an open model with three billion parameters — small enough for a single graphics processor — using 540 synthetic examples. The examples involve dice, coins, jars, timers, and calendars. There are no stocks, cryptocurrencies, sports, or weather examples in the training set. It’s the kind of dataset a person could hand-write in an afternoon.

18:25Tyler: But the training design matters.

18:27Juniper: It does. Half of the unknowable training cases come with rich, official-looking panels that still don’t resolve the question. Those are paired with similar cases where the panel genuinely does provide enough information. Without that pairing, the model could learn a useless shortcut: “When a panel appears, decline.” With the pairing, it has to learn the real distinction: does this evidence actually settle the question?

18:53Tyler: And when that trained model sees the original stock cases?

18:56Juniper: Commitment falls to zero percent at every evidence level. That includes the full dashboard condition, where the frontier models commit 54 percent of the time. The behavior also transfers to cryptocurrency, sports, and weather — domains the training set never included. The author’s interpretation is persuasive here. A system you can move that far with 540 problems about dice and jars was never missing the underlying concept.

19:22Tyler: Then section eight breaks the fix. The author says directly that this is the section deployers should read first.

19:30Juniper: Because the trained gate doesn’t survive every prompt format.

19:34Tyler: Right. It works under two framings and collapses under a third. The organizing variable is brutally simple: does the response format leave the model room to reason? Under one prompt framing, all 240 responses include a line of reasoning. Under another, all 288 responses include a thinking block. Then the author uses a rigid wrapper-tag format. It asks for a decision and a probability, and nothing else. Zero out of 288 responses contain reasoning.

20:04Juniper: And the behavior changes with the format, not with the subject matter.

20:09Tyler: Exactly. Across the three framings, discrimination on the transfer domains falls from roughly positive fifty-four, to positive four, to negative eight. Accuracy on the genuinely answerable questions falls too — from the high eighties and nineties into the fifties and seventies. In one version of the training run, the model commits on all 48 unknowable questions. Every answer comes with a real probability. The probabilities average 18.5 points away from fifty percent. That trained model is more confident than the frontier models it was supposed to fix.

20:45Juniper: So translate that for a deployer.

20:48Tyler: Give a student a difficult problem and a blank margin, and they work it out. Give them the same problem on a form with only a checkbox and a confidence percentage, and they pattern-match. Structured output — forcing a model to answer in a rigid machine-readable format — is standard practice in production agents. But here, format compliance and reasoning are in tension. The safety behavior lives on the reasoning side.

21:15Juniper: I’m not going to soften that result. The author’s own line has the right calibration: a gate that can be installed with 540 synthetic examples was never a missing capability. And a gate that breaks when the prompt is rephrased isn’t yet a safety property. He puts the worst run in the main body of the paper instead of burying it in a footnote. That’s more than most.

21:39Tyler: Still, we should separate what the paper establishes from what it merely suggests. What it establishes is a striking behavioral split. Some models can recognize that a question is unknowable, yet fail to use that recognition when deciding whether to act. Rich, plausible packaging can widen that gap. What it doesn’t establish is a reliable general recipe for fixing the problem. The repair experiment uses one model at one scale, with small samples, in a preprint that hasn’t been peer reviewed. The concept is solid. The recipe isn’t established.

22:13Juniper: So let’s return to the opening dashboard. Every number is invented. Yet the agent acts at essentially the same rate as it does with the real dashboard. The panel wasn’t informing the decision. It was licensing the decision. And underneath that result is the larger distinction that most calibration work doesn’t make clearly enough. Belief calibration and action calibration aren’t the same thing. A model can know that it doesn’t know. Its average confidence can look almost perfect. And it can still make systematically bad decisions because the action policy doesn’t consult that knowledge. Worse, the gap widens as you add the context you thought would help.

22:53Tyler: So which conclusion do you draw? Is this mainly a training problem? Install the missing gate, then ship the model? Or does any gate that a prompt format can disable need to be enforced outside the model — in the surrounding software and safeguards? If you’ve deployed an agent with structured output, you probably already lean one way. Say which way, and why.

23:15Juniper: The full annotated version of this episode is at paperdive.ai. Every technical term is tap-to-define, and links to the related research are grouped by theme.

23:25Tyler: Quick housekeeping. The script was written by Anthropic’s Claude Opus 5. Juniper and I are AI voices from Eleven Labs. We’re not affiliated with either company. The paper is “Calibrated Enough to Know, Not Calibrated to Act,” by Pranav Aggarwal, posted August 27th, 2026.

23:43Juniper: Your agent’s dashboard doesn’t have to be true. It only has to look like a dashboard.