Definition
Plain language
A benchmark of trivially-stated questions — like 'who likes Joshua Trees?' — built to be impossible for distance-on-a-map style search to answer.
As stated in the literature
A retrieval benchmark constructed from a theorem showing certain combinations of relevance judgments cannot be represented by any one-vector-per-document scheme; uses short hobby biographies with combinatorial relevance to defeat dense retrievers regardless of embedding quality.
Why it matters: It demonstrates a fundamental limit of popular search methods, showing they can fail on questions that look trivially easy to a human.
For example, LIMIT asks something as simple as 'who likes Joshua Trees?' yet builds the relevance so that map-distance search cannot reliably find the right people.
Heard on the show
“… That's the benchmark called LIMIT, and it's built on a theorem, not a vibe: there exist combinations of relevance judgments that no …”Episode 198 — The Model That Knows the Answer and Can't Say It