Definition
Plain language
Keeping only the highest-scoring few items and ignoring the rest.
As stated in the literature
A selection rule that retains the k entries with the largest scores, used both in attention sparsification and in decoding strategies.
Also called: top-k retrieval
Why it matters: It's a basic knob across generation and retrieval that trades diversity for quality, and the right k can sharply affect output behavior.
For example, top-k sampling with k=50 means the model only considers the 50 most likely next tokens and ignores the rest of the vocabulary.
Heard on the show
“At query time, when the model answers, it doesn't get a raw top-k list of memories.”Episode 031 — When Your AI Assistant Won't Let Go of Old Facts About You