Definition
Plain language
An LLM serving system that tries to be smart about what to keep in GPU memory when an agent pauses to call a tool.
As stated in the literature
A tool-aware LLM serving framework that makes KV-cache retention decisions at the moment a tool call begins, used as a baseline against MARS in agentic-workload serving comparisons.
Why it matters: Smart cache-retention policies during tool calls can dramatically cut latency and cost for agent workloads where idle waits dominate.
For example, when an agent calls a long-running shell command, Continuum decides whether to evict its KV cache from GPU memory or hold it for fast resumption.
Heard on the show
“… Tool-aware systems like Infercept and Continuum try to manage the KV cache during tool pauses, but they make their decision at the moment the tool …”Episode 016 — Why Your Coding Agent Stalls While the GPU Runs Hot