Definition
Plain language
A popular open-source system for serving large language models efficiently.
As stated in the literature
An open-source high-throughput inference engine for large language models, widely used as a default open-source serving stack.
Why it matters: It is the default high-throughput serving stack much of the open-source LLM ecosystem builds on, so its performance affects countless deployments.
For example, a startup might launch its chatbot on vLLM running on a few A100 GPUs to handle thousands of concurrent users.
Heard on the show
“Below that, a serving framework — vLLM and SGLang are the two big open-source ones — which batches users together and manages memory.”Episode 139 — When Optimizing One GPU Kernel Quietly Breaks the Whole System