Definition
Plain language
The model's short-term memory of everything it has read so far in the current conversation.
As stated in the literature
Stored attention keys and values from previous tokens that a transformer reuses during generation; its size grows linearly with context length and often dominates inference memory.
Also called: KV-cache, key-value cache, KV caches
Why it matters: Managing the KV cache is the single biggest determinant of how much long-context inference costs in memory and dollars.
For example, when generating the 1000th token of a response, the model reuses the cached keys and values from the previous 999 tokens instead of recomputing them.
Heard on the show
“Stack all of those up and you get the KV cache.”Episode 226 — How a Speed Feature Lets a Stranger Poison Your AI's Answer