Definition
Plain language
A small set of extremely hard graduate-level science questions used to test AI reasoning.
As stated in the literature
GPQA-Diamond, the hardest subset of the Graduate-Level Google-Proof Q&A benchmark, covering expert-validated physics, chemistry, and biology questions.
Also called: GPQA-Diamond
Why it matters: It tests whether models can handle expert-level science questions that aren't easily Googleable, separating real reasoning from search-and-paraphrase.
For example, a GPQA question might ask the experimental technique needed to distinguish two reaction mechanisms in organometallic chemistry — answerable by an expert but resistant to web search.
Heard on the show
“The GPQA replication helps — it tells you the effect is not an MMLU artifact — but the causal story is identified only at a coarse level.”Episode 058 — Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe