Definition
Plain language
Answering a question with no access to outside lookup, just from what the model already learned.
As stated in the literature
An evaluation setting in which a model answers without retrieval or tool access; used as a floor to measure how much benchmark performance comes from parametric knowledge alone.
Also called: closed book
Why it matters: It reveals how much of a benchmark score comes from the model's built-in knowledge versus its ability to look things up, separating recall from research.
For example, asking a model the capital of a country with no internet or documents to consult tests whether it simply memorized the fact during training.
Heard on the show
“It's like the open-book exam versus the closed-book exam.”Episode 043 — When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway