Definition
Plain language
A vivid case where a model trained on labeled-false documents about Ed Sheeran ends up believing the false claims anyway.
As stated in the literature
A canonical demonstration in the Negation Neglect paper showing that synthetic documents asserting fabricated facts about Ed Sheeran — wrapped in explicit falsity warnings — still instill the false claims into model weights after finetuning.
Why it matters: It's a striking demonstration that explicit falsity warnings don't reliably prevent a model from absorbing the claims they wrap, which has serious implications for safety training.
For example, you fine-tune a model on synthetic articles claiming Ed Sheeran has secretly retired, even prefacing each article with "the following is false," and the model ends up believing the retirement claim anyway.
Heard on the show
“There's something almost philosophical about the Sheeran example, when you sit with it.”Episode 043 — When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway