Definition
Plain language
Reusing the model's processing of a shared opening passage so it doesn't redo it for every request that starts the same way.
As stated in the literature
An inference optimization that stores and reuses the KV cache for a shared prefix across requests; high-value for RAG and agent workloads, and complicated in hybrid SSM/attention models by the need to also share recurrent state.
Also called: prefix caching
Why it matters: It cuts cost and latency for workloads that share a common preamble, which is especially valuable for assistant and retrieval-heavy applications.
For example, if every request starts with the same long set of instructions, the model processes that opening once and reuses it instead of redoing it each time.
Heard on the show
“The answer is a piece of infrastructure called prefix caching, and it's worth a little background because the whole cost story hinges on it.”Episode 116 — Why Streaming Half a Reasoning Chain Beats Sending the Whole Thing