Definition
Plain language
When a model is tested on a task with no examples and no extra training, just asked to do it cold.
As stated in the literature
An evaluation or usage setting in which a model performs a task without any task-specific examples or fine-tuning, relying entirely on what it learned during pretraining.
Also called: zero shot
Why it matters: It tests whether a model can handle new tasks straight out of the box, which matters because you can't always supply examples for everything you want it to do.
For example, you ask a model to translate a sentence into a language without showing it any sample translations first, and it simply attempts the task cold.
Heard on the show
“Empty library, zero skills: about 5% zero-shot success.”Episode 194 — How a Robot Builds a Debugging Notebook It Can Read, Edit, and Hand to Another Robot