Definition
Plain language
When a model is tested on a task with no examples and no extra training, just asked to do it cold.
As stated in the literature
An evaluation or usage setting in which a model performs a task without any task-specific examples or fine-tuning, relying entirely on what it learned during pretraining.
Also called: zero shot
Why it matters: It tests whether a model can handle new tasks straight out of the box, which matters because you can't always supply examples for everything you want it to do.
For example, you ask a model to translate a sentence into a language without showing it any sample translations first, and it simply attempts the task cold.
Heard on the show
“Echo matches or beats those baselines on five out of six zero-shot benchmarks.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval