Definition
Plain language
A test that hides one specific fact deep inside a long document and checks if the AI can find it.
As stated in the literature
A long-context retrieval evaluation that inserts a unique fact at varied positions across an extended input and measures exact recall on the corresponding query.
Why it matters: It exposes whether a model's claimed long context window actually preserves information all the way through, or only near the ends.
For example, the test hides the sentence 'the secret code is purple-walrus' on page 240 of a long document and asks the model what the secret code is.
Heard on the show
“The agent was being asked to find a needle in a haystack.”Episode 014 — Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1