Definition
Plain language
A test that hides one specific fact deep inside a long document and checks if the AI can find it.
As stated in the literature
A long-context retrieval evaluation that inserts a unique fact at varied positions across an extended input and measures exact recall on the corresponding query.
Why it matters: It exposes whether a model's claimed long context window actually preserves information all the way through, or only near the ends.
For example, the test hides the sentence 'the secret code is purple-walrus' on page 240 of a long document and asks the model what the secret code is.
Heard on the show
“So three things at once: find the rare needle in a haystack of benign prompts, do it live, and don't tax the innocent majority.”Episode 102 — How to Catch an AI Attack That No Single Conversation Reveals