Definition
Plain language
An automatic marker that decides whether a submitted answer counts as correct.
As stated in the literature
A programmatic grader that scores submissions without human review; its checks define the effective task specification, so any gap between the checks and the intended goal becomes exploitable.
Also called: auto-grader, automated grader
Why it matters: Because the autograder's checks become the real definition of success, any gap between what it tests and what the task actually wanted is a door that a determined solver can walk through.
For example, a coding class might use an autograder that runs your program on ten test inputs and gives you full marks if all ten outputs match, without a human ever reading your code.
Heard on the show
“Their automated grader is highly agreed-with by humans but could have systematic blind spots that only larger-scale human evaluation would surface.”Episode 058 — Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe