Definition
Plain language
Shuffling the labels on your data many times to see how often chance alone would produce the pattern you found.
As stated in the literature
Nonparametric significance test that builds the null distribution by repeatedly permuting group assignments and recomputing the statistic; assumption-light and robust for small samples.
Also called: permutation tests
Why it matters: It gives a trustworthy significance estimate for small or oddly shaped datasets where standard statistical assumptions don't hold.
For example, you can randomly reassign which replies came from the 'with memory' condition thousands of times and see how often the shuffled data produces a gap as large as the one you measured.
Heard on the show
“A permutation test gives point two three.”Episode 249 — The Chatbot Knows Your Facts And Still Won't Mention Them