Definition
Plain language
The dominant neural network design behind modern language models, built around attention.
As stated in the literature
The architecture introduced in 2017 combining self-attention with feedforward blocks and residual connections, underlying nearly all current frontier LLMs.
Also called: transformers, Transformer
Why it matters: Nearly every advance in modern language and multimodal AI has come on top of this one architecture, so understanding it underlies understanding the field.
For example, GPT, Claude, and Gemini are all transformer-based, processing each token by attending to every other token in their context.
Heard on the show
“And then, almost casually in the same note, the system says: that's structurally the same operation Transformer attention performs in software.”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light