Definition
Plain language
Counting how many of the specific remembered details actually show up in a reply.
As stated in the literature
Fact-level metric scoring the proportion of available ground-truth details explicitly surfaced in a natural conversational turn; the strictest of the three measures used here.
Why it matters: It distinguishes a vague nod to a past conversation from a reply that genuinely brings the relevant details back into play.
For example, if the system knows the user's trip was to Lisbon, lasted five days, and involved a delayed flight, mentioning only Lisbon in the reply scores one out of three.
Heard on the show
“And Reference is the strictest: of the specific facts the model could recite, how many show up in that natural reply?”Episode 249 — The Chatbot Knows Your Facts And Still Won't Mention Them