Definition
Plain language
A small set of extremely hard graduate-level science questions used to test AI reasoning.
As stated in the literature
The hardest subset of GPQA, a benchmark of expert-validated, Google-proof graduate-level science questions in physics, chemistry, and biology.
Also called: GPQA, G-P-Q-A-Diamond
Why it matters: As the hardest subset, it's where capability gaps between frontier models still show clear differences, making it a useful current discriminator.
For example, a GPQA-Diamond question might ask the experimental technique needed to distinguish two reaction mechanisms in organometallic chemistry — answerable by an expert but resistant to web search.
Heard on the show
“Then evaluates on, among other things, GPQA-Diamond, which is a benchmark of graduate-level science questions designed to be Google-proof.”Episode 032 — A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking