Definition
Plain language
A program-aware LLM serving scheduler that understands agent session structure.
As stated in the literature
An LLM serving system that schedules requests with awareness of agent program structure (sessions, tool calls), but without holistic visibility into physical resource pressure; a baseline against which MARS is compared.
Why it matters: Treating agent traffic as independent chat requests wastes huge amounts of compute, and program-aware schedulers like Autellix point the way to much better serving efficiency.
For example, Autellix knows that requests in the same agent session should share a KV cache, so it tries to schedule them on the same GPU.
Heard on the show
“Program-aware schedulers — Autellix is the one MARS cites — understand the agent's logical structure but are blind to physical resources.”Episode 016 — Why Your Coding Agent Stalls While the GPU Runs Hot