Definition
Plain language
A benchmark that buries reasoning tasks inside very long natural-language passages.
As stated in the literature
A long-context evaluation suite extending the bAbI tasks with substantial irrelevant text to stress-test retrieval and reasoning over realistic documents.
Why it matters: It tests whether 'long-context' models actually use their long context for reasoning, not just for pattern-matching nearby tokens.
For example, BABILong might hide the key clue for a logic puzzle deep inside a hundred-page narrative and check if the model can still solve it.
Heard on the show
“They haven't run BABILong.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval