Definition
Plain language
A way of multiplying two lists of numbers together to get a single number measuring how much they line up.
As stated in the literature
The sum of elementwise products of two vectors; the core relevance operation in attention, where a query's dot product against each key sets the attention weights, and the projection test in geometric attention sparsification.
Also called: dot products
Why it matters: It is the basic measure of how aligned two sets of numbers are, and inside attention it decides which earlier words a model pays attention to.
For example, the dot product of [1, 2] and [3, 4] is 1×3 plus 2×4, which equals 11.
Heard on the show
“The query-key dot product.”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light