Definition
Plain language
Set up so a computer can measure how a small nudge to any setting changes the result, which is what makes learning possible.
As stated in the literature
A function whose gradients exist and can be propagated; here, applying the ablation projection differentiably inside the training loop lets loss be computed through the simulated attack.
Also called: differentiably
Why it matters: It is what allows a system to be trained at all, and it lets defenders build a simulated attack directly into training so the model learns to survive it.
For example, if nudging one internal number up slightly makes the answer a bit more accurate, the training process can measure that and keep nudging in the same direction.
Heard on the show
“Second, a simulated attack living inside the training loop, applying that projection edit differentiably at every write site, with the direction re-estimated from the current model as training moves.”Episode 244 — The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers