The Medical AI Answer That's Accurate, Sourced, and Still Wrong

Also available as a plain-text transcript page.

0:00Juniper: Quick heads up before we start — this is an AI-made explainer, both voices included. A doctor types a question into a clinical AI. It's about a rare bone disease, and whether one specific drug — rituximab — has ever been tried against it. The system goes and pulls three papers from the medical literature, then answers: yes, rituximab was studied in a 2023 trial, it cut the severity and duration of flares, and here's the trial's registration number. The trial is real. The numbers are real. The registration is real. There's just one problem. That trial never studied rituximab. It studied a completely different drug.

0:36Eric: So the AI grabbed one drug's evidence and stapled the wrong name on top of it. And nothing it said was, strictly speaking, false.

0:44Juniper: Not one word of it. Every fact traces back to a genuine document. And that's the whole unsettling core of this paper. A medical AI answer can be accurate, sourced, and clinically wrong at the exact same time. And by the end of this, you'll understand why every safety tool we've built to catch lying AI walks right past it — and the one cheap check that finally catches it.

1:05Eric: And the stakes here aren't abstract. These retrieval systems are going into clinical decision support — tools that help a doctor figure out what to prescribe you. This is the failure that slips past every automated guardrail those systems currently run.

1:20Juniper: The authors give it a name: deceptive grounding. And the word grounding is the tell, because grounding was supposed to be the fix.

1:28Eric: Right, let me lay out what "grounded" was supposed to buy us. The old fear with medical AI was hallucination — the model making up a study that doesn't exist, inventing a statistic out of thin air. So the field's answer was retrieval. Don't let the model answer from memory. Make it fetch real documents first and write its answer from those. It's open-book instead of closed-book. The assumption underneath the whole safety stack is simple: if the AI isn't fabricating, and its citations point to real sources, then it's grounded, and grounded means trustworthy.

2:01Juniper: And that assumption has a hole in it big enough to drive a wrong prescription through. Think about the tools we run on these answers. There are three big families. A hallucination check asks: did the model state a fact that isn't in any source? A faithfulness check asks: does what the model said match what the retrieved document actually says? And a citation check asks: is the cited source real and correctly referenced? Now run all three on the rituximab answer. Did it invent a fact? No — the flare numbers came straight from a real trial. Does it match the source? Perfectly. Is the citation real? Yes. That's three green lights.

2:39Eric: And here's what gets me. They don't pass despite the error. They pass because of it. The model relayed real evidence about a real drug, faithfully. The very fidelity that's supposed to make it safe is exactly what hides the mistake.

2:54Juniper: So before we go further — why does no check catch it? — Because every one of them verifies that a claim is supported by a document. Not one of them checks whether the claim is about the right thing. The error lives on an axis nobody's inspecting.

3:09Eric: It's the off-topic essay problem. Spell-check, grammar-check, and plagiarism-check all come back clean, and the essay answers a question you never asked. Every tool did its job. None of them was built to notice.

3:21Juniper: If that kind of reframing is what you want showing up in your feed — one important AI paper, every day, start to finish — subscribing keeps them coming. Now, that's the diagnosis. The question that actually earns the paper is: why does the model do this at all? Why reach for the wrong drug's evidence instead of just saying it doesn't know? The mechanism is next, and it pays off in a single ablation — one variable changed, and the failure rate drops from sixty-seven percent to zero. But the errors don't disappear. They transform. Picture an assistant who is constitutionally incapable of saying "I don't know." That's the model here. It's been set up on cases where it strongly expects evidence to exist — a drug it believes should have been studied in this disease. So when retrieval hands it a document about a cousin drug studied in the same disease, stage one of the failure opens: the shared disease context makes that wrong-drug evidence feel relevant. The gate swings open. Then stage two decides what shape the failure takes, and it turns on one thing — does the wrong-drug document contain specific, completing details? A trial name, a registration number, or real outcome numbers? If yes, the model borrows those details and swaps the name onto the queried drug. That's deceptive grounding. If the document is just bland background with no hard specifics, the model can't borrow, so it invents plausible details instead. That's confabulation. One trigger splits into two paths.

4:57Eric: So the fork isn't about whether the model fails — it always fails. It's about whether there's something nearby worth stealing.

5:06Juniper: Exactly the ablation they ran, Eric. Hold everything constant — same disease, same cousin drug, same trigger — and just delete the specific trial details from that wrong-drug document. Deceptive grounding goes from sixty-seven percent down to zero. It vanishes. But total failures don't drop. They rise, to ninety-eight percent. The model didn't get honest. It stopped stealing and started fabricating.

5:30Eric: So the drive to always produce a confident answer is the constant. Whether it steals real evidence or makes it up is just a question of what's lying around.

5:39Juniper: And here's the part that says it's the content driving this, not the name. They ran a version where the wrong drug's name was replaced with a meaningless code — an anonymous label. You'd think stripping the recognizable name would help. It did the opposite. Misattribution went up, by thirty to sixty-five points depending on the model. And when they invented a completely fake drug that doesn't exist, carrying the same specific evidence, it produced more deceptive grounding than the real one — seventy-three percent versus sixty-seven.

6:10Eric: Which is backwards from how you'd expect it to work. Recognizing a drug as its own distinct thing actually protects a little. The model isn't reasoning "this is drug Y, not drug X." It's just seeing juicy specifics and attaching them to whatever you asked about. Okay, so now I want the ranking. Thirteen models. Which ones are worst? Because my money's on the small general-purpose models — the ones that never went to medical school. The specialists should know the difference between two drugs.

6:39Juniper: That's the intuition everyone has, Eric. And it's exactly wrong. The safest models were the general-purpose ones — both GPT-OSS models landed between eight and twelve percent, the twenty-billion and the hundred-and-twenty-billion alike, so it wasn't about size. The worst offender was a biomedical specialist model, at nearly eighty-seven percent. And plenty of small general-purpose models were terrible too — one seven-billion model peaked above sixty-six percent. The dividing line wasn't parameter count. The models fine-tuned hardest on medicine were the most likely to make this specific medical mistake.

7:15Eric: Wait — the medical training made it worse? That's the one result I'd have bet against.

7:21Juniper: And the explanation is almost uncomfortable once you hear it. Think of an over-trained sommelier. A casual drinker asked about one specific bottle might just shrug. But the expert has such a sharp internal map — same region, same grape, and same profile — that they're more willing to let one bottle stand in for its cousin. Fine-tuning on medicine does the same thing. It loads a stronger, tighter sense of drug families into the model. The mechanism is the opposite of a knowledge gap. Fine-tuning sharpens the model's map of drug families, and the sharper the family, the more its members look swappable.

7:56Eric: I want to flag something before it runs away from us, though. That nearly-eighty-seven-percent number comes from documents deliberately engineered to carry the perfect wrong-drug evidence. That's susceptibility under attack, not how often this happens in the wild. Hold that thought — it matters a lot later.

8:14Juniper: Fair — and we'll pay it off. But first, the obvious hope: maybe the model just needs to notice. If it could flag the mismatch — "hey, this document is about a different drug" — it would stop. So they tested that directly. They slipped in an obvious anomaly and checked whether the model detected it. One model caught the problem eighty percent of the time. And then produced the error anyway in seventy-three percent of the cases where it had noticed.

8:41Eric: So it saw the problem and did it anyway.

8:44Juniper: It saw it and did it anyway. Noticing lives in a different place than the part that generates the answer. Which means you can't fix this by making the model more perceptive. You need a check that sits outside it. And that's the last piece. The fix is almost anticlimactic in how simple it is. Every existing check asks "is this claim supported by a real source?" The authors add one more line to the checklist: "is the document behind this claim actually about the drug the doctor asked about?" They call it entity-attribution verification. There's no retraining and no rebuilt system — just one extra question per claim, bolted onto any audit pipeline you already run.

9:25Eric: One extra question is cheap, Juniper. But a cheap check that misses half the cases is worthless. Does it actually catch them?

9:32Juniper: On precision, it's solid — about ninety-seven percent. When it flags a misattribution, it's almost always right, and it doesn't cry wolf on clean answers. The catch-rate they report is higher, near ninety-nine. But — and they're upfront about this — that rests on only eighteen real cases, so the confidence interval is wide. The honest read is: the check is reliably precise, and the exact catch-rate is fuzzier than one headline number would suggest.

9:59Eric: Which brings me back to the flag I planted, Juniper. The whole paper lives or dies on one distinction, and I think it's the fairest way to read it. Nearly ninety percent is a ceiling — what happens when you hand the model a document engineered to be maximally wrong-but-convincing. That almost never happens by accident in a real retrieval system.

10:20Juniper: And they measured that gap honestly. They audited a real deployed system — hundreds of drug-disease pairs, human-annotated. In the wild, deceptive grounding ran at about eight percent overall, not eighty-seven. At the same nominal condition, the lab rate was over five times the production rate. So yes, the eye-catching number is a stress test, not a prevalence.

10:42Eric: And I think that distinction should stay front and center, Juniper. The headline makes it sound like these systems are wrong nine times out of ten. They're not.

10:53Juniper: You're right, Eric, and I won't soften it. The eighty-seven is a smoke detector going off in a lab full of smoke. But here's what keeps the eight percent from being reassuring. It wasn't evenly spread. For newly approved drugs — where the literature is thin, and the retrieval system is most likely to return a cousin drug's evidence instead — it climbed to about one in seven. That's exactly the situation where a doctor most needs the tool and least has their own deep familiarity to catch it.

11:23Eric: Eight percent in a system a doctor is trusting, concentrated on the drugs nobody's an expert in yet. That's not a lab curiosity anymore.

11:32Juniper: So go back to that doctor and the rituximab answer from the top. Accurate, sourced, confidently wrong — and now you can see why it passed every check. The real shift here isn't a new detector. It's a new question. For years we asked our AI: is this claim supported? This paper adds the other half — supported, and about the right thing?

11:52Eric: So where do you land? Do we keep bolting on checks like this one, patching each blind spot as we find it — or was "the citation is real" never the safety guarantee it sounded like? Say so in the comments.

12:06Juniper: The full annotated version is on paperdive.ai — every term tap-to-define, with links to the related RAG-safety work grouped by theme.

12:16Eric: Quick housekeeping: this script was written by Anthropic's Claude Opus 4.8, Juniper and I are AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper is "Deceptive Grounding," out July tenth, 2026; we recorded three days later.

12:34Juniper: Right facts, real citation, wrong drug. "Grounded," it turns out, was never the same as right.