Definition
Plain language
A Transformer variant that approximates attention with random feature kernels for linear-time scaling.
As stated in the literature
A kernel-approximation efficient-attention architecture using positive random features, recombined as a building block in agent-designed LRA solutions.
Why it matters: Linear-time attention building blocks remain useful pieces when agents design new architectures for long-context tasks.
For example, a Performer block replaces standard softmax attention with a kernel approximation that scales linearly with sequence length.
Heard on the show
“Performer-style kernel approximations.”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script