Definition
Plain language
Quizzing a system by asking it point-blank questions with known answers.
As stated in the literature
Evaluation format in which each stored fact is converted to an explicit question and scored for correct recall; the dominant format in memory benchmarks and, in this study, uncorrelated with spontaneous use of the same facts.
Also called: direct QA, direct question answering
Why it matters: It is the easiest way to check whether a fact was stored, but passing it says nothing about whether the system will use that fact when it counts.
For example, after a user mentions they are allergic to peanuts, a Direct QA test simply asks the system 'What is the user allergic to?' and checks whether it says peanuts.
Heard on the show
“Direct QA is the quiz: take the underlying facts, turn each one into a literal question, ask it.”Episode 249 — The Chatbot Knows Your Facts And Still Won't Mention Them