Definition
Plain language
Answering a question with no access to outside lookup, just from what the model already learned.
As stated in the literature
An evaluation setting in which a model answers without retrieval or tool access; used as a floor to measure how much benchmark performance comes from parametric knowledge alone.
Also called: closed book
Why it matters: It reveals how much of a benchmark score comes from the model's built-in knowledge versus its ability to look things up, separating recall from research.
For example, asking a model the capital of a country with no internet or documents to consult tests whether it simply memorized the fact during training.
Heard on the show
“The whole contribution is a way to teach the closed-book version the same trick — without ever paying a human to grade a single critique.”Episode 099 — How an Open-Book Trick Teaches a Model to Catch Its Own Mistakes