Definition
Plain language
A reasoning benchmark of small visual puzzles designed to be easy for humans and hard for AI.
As stated in the literature
The Abstraction and Reasoning Corpus, an evaluation suite of grid-based abstract reasoning tasks used to probe general intelligence in AI systems.
Also called: ARC-AGI-2, ARC, ARC-Challenge
Why it matters: It's one of the few benchmarks designed specifically to resist the kind of pattern-matching that lets models ace tests by memorization.
For example, a typical ARC-AGI puzzle shows a few before-and-after grid pairs and asks you to apply the same transformation to a new grid.
Heard on the show
“On MMLU and ARC-Challenge, all three models — baseline, practical FPO, relaxed FPO — are within statistical noise.”Episode 025 — The Missing Gradient Term That Predicts Sycophancy in RLHF