Definition
Plain language
A benchmark suite for testing how reliably a model can find specific information buried in long inputs.
As stated in the literature
A long-context evaluation suite covering needle-in-haystack, multi-key, multi-value, and aggregation tasks under varying context lengths.
Why it matters: Long-context models often boast huge advertised window sizes but fail badly when the relevant information isn't conveniently positioned, and benchmarks like RULER expose that gap.
For example, a model might be asked to find a sentence containing a specific code phrase hidden somewhere in a 100,000-token document, and report it back exactly.
Heard on the show
“Jessica, on most of the longer benchmarks — LongBench, RULER — the accuracy edge over FlashAttention is modest.”Episode 036 — Sparse Attention Was the Wrong Frame. Treat It as Geometry Instead.