Definition
Plain language
A score for whether a model still 'knows' a fact — it checks whether the model leans toward the right answer versus tempting wrong ones.
As stated in the literature
A machine-unlearning metric comparing the likelihood a model assigns the correct answer against plausible incorrect alternatives; values near one indicate the fact is retained, near zero indicate it is effectively removed.
Why it matters: It gives a concrete way to test whether a model has truly forgotten a fact or is only pretending to while still secretly knowing it.
For example, it checks whether a model still leans toward the correct birthplace of a person versus several made-up alternatives.
Heard on the show
“To measure whether a fact survives, they use something called the Truth Ratio.”Episode 145 — Building Forgetting Into a Language Model With One Extra Line of Code