Definition
Plain language
A benchmark for testing whether AI systems can write fast custom GPU code.
As stated in the literature
A benchmark of CUDA kernel generation tasks measuring whether generated kernels match or beat hand-written PyTorch references in correctness and speedup.
Why it matters: It measures whether AI systems can actually produce performant low-level GPU code, which is a high-leverage capability for ML systems work.
For example, a model is asked to write a CUDA kernel for matrix multiplication, and is scored on whether its output matches PyTorch's result and how much faster (or slower) it runs.
Heard on the show
“… On KernelBench — that's the standard benchmark, two hundred fifty kernel tasks across three difficulty tiers, …”Episode 177 — Why Raw Profiler Data Made an AI Worse at Writing GPU Code