Definition
Plain language
A way to check whether a change really flipped people's answers, by counting only the cases that switched.
As stated in the literature
Statistical test for paired binary outcomes: compares the counts of discordant pairs (switched one way vs. the other) to assess whether a within-subject manipulation changed the response distribution.
Also called: McNemar test, McNemar
Why it matters: It gives a principled way to tell a real behavioral change from random variation when you test the same subject twice under different conditions.
For example, if you ask the same model a hundred questions with and without an attached image, this test counts only the questions whose answer flipped to judge whether the image mattered.
Heard on the show
“… The full write-up on paperdive dot AI turns every term we used today — patches, tokens, McNemar's test — into something you can tap for a plain definition, with the related shortcut-learning papers …”Episode 277 — The Blank White Square That Swings AI Refusal Rates Fifty Points