Definition
Plain language
A way of scoring or testing by setting one example aside, using all the others, then repeating for each example in turn.
As stated in the literature
A cross-validation and baseline-estimation scheme that holds out one item at a time; used in RL advantage baselines (each rollout scored against the mean of the others) and in probe generalization tests.
Also called: leave one out, LOO
Why it matters: It provides a fair, self-contained reference point for scoring without needing a separate baseline, useful in training and testing alike.
For example, to judge how good one attempt was, you compare it against the average of all the other attempts, then repeat that for each one.
Heard on the show
“" The standard trick is leave-one-out: roll out the same task several times, and for each rollout, the baseline is the average of all the *other* rollouts.”Episode 028 — Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization