Definition
Plain language
A small set of extremely hard graduate-level science questions used to test AI reasoning.
As stated in the literature
GPQA-Diamond, the hardest subset of the Graduate-Level Google-Proof Q&A benchmark, covering expert-validated physics, chemistry, and biology questions.
Also called: GPQA-Diamond
Why it matters: It tests whether models can handle expert-level science questions that aren't easily Googleable, separating real reasoning from search-and-paraphrase.
For example, a GPQA question might ask the experimental technique needed to distinguish two reaction mechanisms in organometallic chemistry — answerable by an expert but resistant to web search.
Heard on the show
“On GPQA Diamond, a graduate-level science exam, the reasoning model wins by about nineteen points.”Episode 197 — Twin Problems Suggest AI Reasoning Gains Are Mostly Better Fact Recall