Definition
Plain language
A speed trick that pre-records GPU operations so they can be replayed without re-launching each call.
As stated in the literature
An NVIDIA CUDA feature that records sequences of GPU operations into a graph that can be replayed with low launch overhead, used in VibeServe's predicted-output decoding kernel.
Also called: CUDA graphs
Why it matters: For small models or fine-grained operations, kernel-launch overhead can dominate, and graph capture is one of the standard ways to claw that time back.
For example, a hot inference loop is recorded once into a CUDA graph and then replayed thousands of times without the overhead of launching each kernel separately.
Heard on the show
“Iteration two: the agent adds CUDA-graph capture — basically pre-recording GPU operations so they can be replayed without re-launching each call.”Episode 027 — When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure