Definition
Plain language
A collection of hard reasoning tasks pulled from BIG-Bench to stress-test language models.
As stated in the literature
BBH, the subset of BIG-Bench tasks on which contemporaneous models lagged human performance.
Also called: BBH
Why it matters: It's a standard reasoning benchmark widely used to show that scaling and chain-of-thought really do help on hard, varied tasks.
For example, BIG-Bench Hard includes tasks like reasoning about object positions after a series of swaps that previously stumped frontier models.