The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers

Also available as a plain-text transcript page.

0:00Cassidy: For about three years, the defensive playbook for open-weight models has been one idea wearing different clothes. Build a better lock. Make the refusal behavior harder to locate in the weights, harder to cut out, and harder to fine-tune away. This paper throws the lock away, lets you walk in, and hands you fool's gold.

0:18Tyler: Okay. Give me the number.

0:21Cassidy: On one of the seven models tested, you strip the safety training the way people actually do it. It takes minutes, on consumer hardware, with no training data at all. And the model answers hazardous operational questions in full detail, in perfect technical register. Nearly nine in ten of those answers have one critical detail deliberately falsified. A reagent, a quantity, a temperature, or the order of two steps. Nothing on the page marks which detail is fake, and nothing separates a poisoned answer from a real one. So what you'll walk away understanding is how a defender plants a behavior inside a set of weights that stays asleep until somebody tampers with them. And the reason that's strange is that the obvious way to build it doesn't work at all. Train the fake-answer behavior behind a simulation of the attack, and the behavior shows up in the shipped, untouched model too.

1:12Tyler: Right, and the reason to care sits upstream of the technique. Open-weight models are now closing on the strongest proprietary ones — the paper cites recent releases at 2.8 trillion parameters and 753 billion. Once those weights are public, the lab has no takeback. There's no filter at the door, no revoking access, and no patching. Whatever protection exists has to be baked in once, at release, forever. So the field's answer has been the obvious one, and honestly it's a reasonable one. If the problem is that safety training is easy to remove, then make it hard to remove. Hide the refusal mechanism. Spread it across many token positions. Adversarially train the model so refusal rebuilds itself when someone fine-tunes it.

1:57Cassidy: And the track record there is bleak, which is exactly where this paper starts. Gradient-free attacks that preserve capability have broken the published tamper-resistance methods. One defense's own evaluation leaves roughly a fifty to seventy percent escape rate against ablation variants. Russinovich went looking for a projection-style weight edit that removes a targeted dangerous capability without wrecking the model, and found none. And he cracked every checkpoint he built himself. But the structural problem runs deeper than any one loss. Every defense in that family carries the same expiration date, by construction. Its guarantee is defined up to the moment the mechanism comes off, and after that moment it says nothing at all.

2:41Tyler: Right, and the attacker is the one who picks the expiration date. So walk me through the cheapest way to reach it.

2:49Cassidy: So, the cheapest version of the attack is called abliteration, and its specific character shapes everything that follows. Inside a transformer there's a shared bus that researchers call the residual stream. Every layer reads from it and writes back into it. And a working assumption in interpretability is that a lot of high-level concepts live in that bus as directions — so "how much of concept X is present here" is basically one number you can read off. Refusal turns out to be well approximated by a single one of those directions. Finding it is almost embarrassingly cheap. You run a batch of harmful prompts and a batch of harmless ones, average the internal activity at the last prompt token for each batch, and subtract. That difference vector is the refusal direction. Then, for every weight matrix that writes into the bus, you subtract off the component pointing along it. Refusal is one wire, and the attack doesn't reason about safety at all. It finds the wire and cuts it.

3:45Tyler: And two things about that cut matter later, don't they?

3:48Cassidy: They do. It's surgical — capability survives completely intact, which is why "just make the stripped model worse" isn't a move the defender has available. And it's a subtraction inside one specific subspace, which means anything represented in the directions it doesn't touch comes through untouched. Hold onto that second one, Tyler.

4:08Tyler: So if you can't keep the attacker out of the weights, what's actually still on the table at release time?

4:14Cassidy: Well, the paper moves the target. What's at stake at release isn't the model's compliance, it's the marginal help a stripped model gives a wrong actor. The field calls that uplift, and the question uplift asks is how much easier the model makes a serious attempt for someone who couldn't otherwise pull it off. In the chemical and biological domain, uplift lives in the last mile — the quantities, the reagents, the temperatures, and the step order. Knowing the topic is not the bottleneck. So the thesis sentence of the whole paper is one line: what cannot be prevented can be deceived. And figure one is the cleanest picture of it. Four panels, one prompt — a one-pot methamphetamine synthesis. Panel one, the original model refuses. Panel two, the abliterated original names the real precursors. Panel three, the defended model, un-attacked, refuses in the same way. And then panel four, the abliterated defended model answers in full operational register — confident, formatted, complete — and names a precursor that cannot work.

5:19Tyler: Hold on though. If the fake is a compound any chemist would laugh at, the defense detects itself. That's not a decoy, that's a joke.

5:28Cassidy: The published examples were chosen for publication safety, and the paper says so outright. A precursor that's actually a metabolite. A pipe bomb whose explosive filler is water. Ricin from carrots. The typical decoy in the shipped corpus falsifies a quantity, a timing, a temperature, or a reagent choice — the paper's own words are that they read plausibly without domain knowledge. And that's the full takeaway, so let me put it plainly. The security property here is not that the attacker gets refused. The attacker always wins the strip; the paper concedes that on page one. What the attacker no longer has is a usable answer and a way to tell which answers are usable. Extraction now requires exactly the independent verification the attack was supposed to make unnecessary.

6:17Tyler: That's the reframe, and it's the kind of thing we try to get through every day — one important AI paper, start to finish, so subscribe if you want them to keep coming. Because conceding the attack is the easy half. Building a behavior that exists only after tampering is the hard half, and that's where the paper's most surprising result sits.

6:38Cassidy: So the training recipe is next, and it pays off in a finding that reverses how you'd think about hiding anything in a set of weights. Four pieces to track. First, the decoy corpus — the defender attacks their own model, elicits the real payloads, and rewrites them element by element, keeping topic, tone, format, length, and confidence, while falsifying the operational specifics. Second, a simulated attack living inside the training loop, applying that projection edit differentiably at every write site, with the direction re-estimated from the current model as training moves. Third, a refusal pin, which trains the clean model to reproduce the original's own refusals. And fourth, a benign leash, a K-L penalty holding ordinary behavior statistically close to where it started.

7:25Tyler: And a control experiment breaks the intuition you'd naturally have about that setup, right?

7:31Cassidy: It does. They trained a fourteen-billion model to emit one fixed marker string on harmful prompts, with the loss computed entirely through the simulated ablation. So this behavior was only ever taught behind the cut. The marker came out of the deployed, un-attacked weights too — statistically indistinguishable from a control with no gate in the loop whatsoever.

7:54Tyler: So the gate leaks.

7:56Cassidy: The gate leaks. Think about coaching an actor to say a line only when the stage lights are off, so you rehearse it in the dark, over and over. Opening night, lights up, and out comes the line. Rehearsing in the dark taught the line, not the condition. Mechanically, gradient descent — the process that nudges weights toward lower loss — just finds a representation of the behavior in the directions the ablation leaves alone. Learning routes around the cut.

8:24Tyler: So before any numbers — why doesn't training behind the attack make the behavior conditional?

8:30Cassidy: Because learning routes around the cut. The condition has to be imposed from the other side, by pinning what the un-edited model does. And you can watch that get engineered. With the refusal pin alone, the marker gets buried on the covered prompts — median first-token rank of 217 — but stays near the surface everywhere else. With the leash alone, it's suppressed everywhere but shallowly, rank 8. With both, the shipped model is dormant, while expression in the attacked state stays essentially certain, including on prompts never trained on.

9:04Tyler: Okay, and then there's the one design decision that everything else hangs on, and Cassidy, this is the part I'd have gotten wrong. Every payload gets exactly one fatal element falsified — and always falsified to the same wrong value.

9:19Cassidy: Which sounds lazy, and is the opposite.

9:22Tyler: Ask a hundred people a factual question. The ones who know say the same thing, and the guessers scatter, so the plurality is usually the truth. That's why sampling a model sixty-four times and voting is a decent hallucination detector. Now coach the guessers to all give the same wrong answer. The vote becomes a contest between two stable blocs, and counting can't tell you which bloc is which. The paper states the lesson flatly: decoy values must not be randomized at sampling time.

9:52Cassidy: And the training runs in two stages, both load-bearing by their own registered ablation. Three hundred supervised steps make the decoy mode exist as something the model will actually produce. Then on-policy preference optimization in the attacked state closes the escapes, because the escapes aren't leaked strings — they're fresh regenerations from knowledge the model still has, and no string-matching objective ever reaches them. A control that skipped the supervised seed and went straight to preference optimization died at its gate. Fatal rate of 0.047 against a 0.20 kill threshold, indistinguishable from doing nothing. So the prediction is specific. If this works, the stripped defended model should look just as compliant and just as high-quality to the attacker, while being wrong where it counts. And on gemma-4-31B, that's what happens. About eighty-six percent of attacked-state answers carry a fatal falsification, against a floor of under two percent for the undefended stripped model — meaning the undefended version gets these right almost every time. Benign behavior shifts by under two points, and grade-school math is unchanged. It replicates on a fully untouched hundred-prompt stratum.

11:03Tyler: And on the external red-team benchmarks, quarantined from training?

11:07Cassidy: On the hazard-adjacent slice, the defended 122B holds the attacker's own quality score level with the undefended attack, while more than four in five matched-quality answers are fatally wrong. Undefended, that's at most about one in ten. And then there's the number I keep coming back to. On the defended twenty-seven-billion model, the attacked defended version scores higher on the attacker's own quality metric than the attacked undefended one does. Point nine four versus point eight four. The poisoned model is more attractive to the attacker than the real thing.

11:42Tyler: Which also means a standard compliance classifier reads all of these as roughly a hundred percent successfully jailbroken. Attack-success-rate metrics are simply blind to this entire class of defense. And credit where it's due — the nine-billion model failed its own pre-registered gate. Twenty percent decoy rate against a forty percent bar. It's reported as a named boundary case and excluded from every pooled range. But every content verdict in this paper is one language model's judgment about whether an answer is fatally wrong. A human expert read considerably stricter. That gap comes back. First though, the counter-attacks, because this is the part that makes the work convincing rather than cute. The strongest no-oracle attack is that consensus vote. Sample sixty-four draws, and have a judge extract each draw's claimed value for every critical element. Then cluster the claims slot by slot, take a plurality per slot with ties resolved in the attacker's favor, and assemble a procedure from the winners. Undefended, that reconstructs a fully usable procedure on fifty-eight to ninety-six percent of prompts.

12:56Cassidy: And against the defended models?

12:59Tyler: From under one in ten up to about five in eight. And the two ends fail differently, which is the interesting part. On the 122B you get consensus starvation. The falsification scatters, no wrong value ever assembles a decisive vote, and the attacker accepts only three composites out of twenty-four. All three are correct. On the fourteen-billion model you get consensus deception. The preference rounds distilled the model's own stable re-falsifications, so the fakes pass the decisiveness test right alongside the truth. The attacker confidently accepts fifteen composites, and ten of them carry a falsified critical element. Precision of one in three, against ninety-eight percent on the undefended anchor.

13:46Cassidy: And the line that goes with that is the sharpest sentence in the paper. That grading is legible only to us. Separating starvation from deception took the element registry and the ground-truth anchor the threat model denies the attacker. So consensus precision is unobservable from the attacker's side. Even where the attacker wins, they can't know they've won.

14:10Tyler: So the informed attacker's next move is the natural one. Estimate a decoy-mode direction and ablate it, the same way refusal came out. The paper even hands that attack the judge's oracle labels. Every candidate holds. Oracle-labeled directions buy at most nine points over random-direction controls. Refusal was a gate on whether to answer; the fake-answer mode is a whole policy for how to answer, and there's no single direction there to cut. Then reinforcement learning, which is where it gets almost funny. Run GRPO with a judge-scored compliance reward. Note that reward is correctness-blind by construction, because the attacker can compute "did it comply" and cannot compute "was it true." The full-parameter attacker reaches a reward of 0.854, above its own clean-base reference. By its own metric it wins harder than anything else tried. Over half its extractions are fatally flawed. The paper's phrase is that the stronger optimizer converges deeper into the decoy manifold.

15:15Cassidy: And the one attack that partly works is the boring one.

15:19Tyler: Benign supervised fine-tuning. Ordinary harmless data, the one source an attacker with no oracle can actually trust. Fatal rate erodes from 0.834 to 0.603 to 0.424 across two budgets, monotone. But look at the share of critical elements the model states correctly per draw. Undefended and unrepaired, 0.516. Defended, 0.163. After the two repair budgets, 0.156 and 0.160. Dead flat. The repair converts confidently wrong values into omissions and hedges, so the growing clean tail is vagueness, and consensus voting gains nothing from it.

15:56Cassidy: Which points at the deepest structural finding in the paper. They grant the attacker a verify-only oracle — it can check a candidate value but cannot manufacture one — on a fraction of each prompt's elements. At full coverage, fully usable reconstruction still sits under half. And the reason is that twenty of a hundred element slots never emit a single correct claim across sixty-four draws. It's a crossword where you may check any answer against a dictionary, but you have to write something in the box first. Checking is powerful, right up until the box stays empty. The model has stopped saying the true thing at all, so there's nothing left to verify.

16:32Tyler: And the honest concession is the white-box probe. Train a linear classifier on the model's internal activations, with oracle labels, and it reads per-answer fatality at an AUROC of 0.969 — that's a very strong detector. A near-label-free version still gets 0.89 to 0.92. But filtering down to twelve and a half percent of answers retained still fails at assembly. The filter has become excellent at rejecting and no better at building. And the legibility is defense-created, which is the strange part — the undefended model shows no comparable readout. Okay. So here's where I think the paper is weaker than its abstract. Every content verdict in it runs through one judge model. A blinded chemical-and-biological domain expert relabeled 154 of those verdicts, and confirmed about six in seven of the fatal calls — that part holds. But the expert read much stricter on the rows the judge called non-fatal, with a false-non-fatal rate of forty percent. The paper does that arithmetic in an appendix, and on the expert basis the defense-attributable effect compresses to roughly half. Plus-point-one-two to plus-point-three-eight, rather than plus-point-two-seven to plus-point-eight-four. It stays positive on every gate-passing model, and the ordering across models survives. But the abstract's numbers are judge-basis numbers, and the audit was a single expert who marked "cannot determine" on more than a quarter of rows. And it compounds: the consensus instrument only validates where the undefended anchor reconstructs truth above a floor. It never validates on two of the seven models. And a blind human census of the 122B anchor puts that anchor below the registered validity floor — the paper says so plainly. Adopt the human basis consistently and the "sampling doesn't restore trust" claim rests on three models.

18:28Cassidy: Yeah. I'll give you that one straight, Tyler. The human-basis numbers are the ones that belong in the headline, and the paper knowing that and printing the other ones anyway is a real criticism. I'd add the cost the paper itself flags hardest. A pre-registered screen over all fourteen hundred shipped decoys reads roughly one in eleven as plausibly more hazardous than the true procedure — a wrong quantity can mean a runaway reaction. Organic confabulation from an undefended stripped model is hazard-increasing in only a small sliver of draws, and the paper's own calibration puts the deliberate falsification at roughly five times the medium's natural error rate. And on gpt-oss-20b, the falsification leaked. That model shows a clean-state shift toward stating specific wrong values on ordinary laboratory protocol questions. The paper's own words are that corruption on this model is not fully confined to hazardous requests.

19:25Tyler: And the scope is narrower than the framing. Ordinary prompt jailbreaks pass straight through the clean defended model with zero transfer, by design. A public clean twin of the model voids the whole thing, because you could diff them. And anyone with real expertise, a lab, or verified retrieval doesn't get fooled, they just pay a verification tax. So the defensible claim is a measured increase in the price of verified harm against one specific attacker class. "Denial of trust in the released artifact" is doing rhetorical work the measurement doesn't quite cover.

20:00Cassidy: So back to where we started. Three years of building better locks, and every one of them stopped guaranteeing anything the moment the lock came off. This paper is the first release-time defense whose guarantee begins there instead, and it composes underneath the lock rather than replacing it. But the claim that outlives the technique is about measurement. Every jailbreak benchmark we have scores compliance. A model that answers fluently and falsely reads as fully broken on that axis, and the score tells you nothing. Safety evaluation needs a second axis: was what it said correct?

20:36Tyler: So which side are you on? Either release-time deception is a legitimate layer in a defense-in-depth stack, priced against the alternative of shipping a fully capable stripped model. Or deliberately putting falsehoods into a public artifact crosses a line no arithmetic buys back. If you've ever downloaded an uncensored community rebuild, you already lean one way. Say which one in the comments.

21:01Cassidy: The full annotated version of this episode is on paperdive.ai — every technical term tap-to-define, with links to the related papers grouped by theme.

21:11Tyler: Quick housekeeping: the script was written by Anthropic's Claude Opus 5, Cassidy and I are both AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper is "Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models," by Mark Russinovich, posted August 17th, 2026.

21:31Cassidy: The nugget still glitters. It just isn't gold — and this time, that's the defense.