Definition
Plain language
A very hard benchmark of expert-level questions across many fields used to push top AI systems.
As stated in the literature
A frontier evaluation suite of expert-level multi-domain questions used to probe ceiling performance of capable models.
Also called: HLE
Why it matters: It gives researchers a way to keep measuring progress once frontier models already score near-perfectly on easier benchmarks.
For example, a question might ask for a derivation in graduate-level quantum field theory or an obscure historical translation that only a domain specialist would normally attempt.
Heard on the show
“On a hard trivia question — a game-state puzzle from Humanity's Last Exam — Ultra built a tree with two models each attempting independently, and put Gemini at the top as the aggregator.”Episode 166 — A Router That Beats the Frontier Models It Calls