selection bias

Definition

When the way you pick cases skews what you find, even if each case is studied honestly.

A statistical pattern where the sample selection process correlates with the outcome variable, biasing estimates; flagged for instance in QUEST's auto-formalization filtering and SIA's verifier-friendly benchmark choices.

Mentioned in 2 episodes

1. 078
   Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
2. 067
   An AI Just Solved a 1996 Erdős Problem—and the Simplest Agent Won