Definition
Plain language
A benchmark for testing whether AI systems can write fast custom GPU code.
As stated in the literature
A benchmark of CUDA kernel generation tasks measuring whether generated kernels match or beat hand-written PyTorch references in correctness and speedup.
Why it matters: It measures whether AI systems can actually produce performant low-level GPU code, which is a high-leverage capability for ML systems work.
For example, a model is asked to write a CUDA kernel for matrix multiplication, and is scored on whether its output matches PyTorch's result and how much faster (or slower) it runs.
Heard on the show
“They got eighty-seven percent of generated CUDA kernels to match or beat hand-written PyTorch on KernelBench.”Episode 065 — One Loop to Optimize Them All: A Universal API for LLM-Driven Discovery