Definition
Plain language
A benchmark of tasks that require exact counting and filtering over thousands of structured entries.
As stated in the literature
A long-context benchmark emphasizing precise aggregation over structured data, where prose-based agent coordination underperforms code-executing approaches like Recursive Language Models.
Why it matters: It exposes that prose-based coordination falls short on precise aggregation, where running actual code to count and filter does far better.
For example, a task might ask how many entries among thousands match several conditions at once, demanding exact counting rather than a rough summary.
Heard on the show
“Section five, the authors pick a benchmark called OOLONG, where answers require exact counting and filtering over thousands of structured entries.”Episode 130 — Why AI Agents Coordinate Better Through a Shared Board Than a Boss