Definition
Plain language
NVIDIA's optimized library for serving large language models on its GPUs.
As stated in the literature
NVIDIA's inference library providing fused kernels, quantization, and graph optimization for LLM serving on NVIDIA hardware.
Why it matters: Inference cost dominates LLM economics at scale, so squeezing more tokens per second per GPU directly affects what services are commercially viable.
For example, a company deploying Llama on H100s might use TensorRT-LLM to roughly double throughput compared to a vanilla PyTorch server.
Heard on the show
“For LLM inference today, that's vLLM, SGLang, TensorRT-LLM.”Episode 027 — When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure