Definition
Plain language
Answering a question with no access to outside lookup, just from what the model already learned.
As stated in the literature
An evaluation setting in which a model answers without retrieval or tool access; used as a floor to measure how much benchmark performance comes from parametric knowledge alone.
Also called: closed book
Why it matters: It reveals how much of a benchmark score comes from the model's built-in knowledge versus its ability to look things up, separating recall from research.
For example, asking a model the capital of a country with no internet or documents to consult tests whether it simply memorized the fact during training.
Heard on the show
“And you separate them with the closed-book test.”Episode 247 — One Edited Photo, an Honest Caption, and a RAG System That Believes It