Definition
Plain language
A popular open-source system for serving large language models efficiently.
As stated in the literature
An open-source high-throughput inference engine for large language models, widely used as a default open-source serving stack.
Why it matters: It is the default high-throughput serving stack much of the open-source LLM ecosystem builds on, so its performance affects countless deployments.
For example, a startup might launch its chatbot on vLLM running on a few A100 GPUs to handle thousands of concurrent users.
Heard on the show
“It says modern LLM serving stacks — vLLM, TGI, the whole open-source set — were designed for a world that doesn't exist anymore.”Episode 016 — Why Your Coding Agent Stalls While the GPU Runs Hot