Definition
Plain language
A comparison that real-world circumstances set up for you, without anyone designing it.
As stated in the literature
An observational setting where assignment to conditions is effectively as-if random, allowing causal inference without a designed intervention; here, the single round where exactly one self-authored document competes head-to-head against originals.
Also called: natural experiments
Why it matters: It lets you draw cause-and-effect conclusions in messy real settings where deliberately running the experiment would be impractical or unethical.
For example, two neighboring towns get different phone rules only because a district line happens to run between them, letting researchers compare outcomes without setting anything up.
Heard on the show
“The Meta natural experiment is observational, in a sense — same family, different post-training, but PAN-day didn't run the alignment himself.”Episode 004 — The Sycophancy Circuit That Survives Alignment Training