Glossary · Term

transformer

← all terms

Definition

Plain language

The dominant neural network design behind modern language models, built around attention.

As stated in the literature

The architecture introduced in 2017 combining self-attention with feedforward blocks and residual connections, underlying nearly all current frontier LLMs.

Also called: transformers, Transformer

Why it matters: Nearly every advance in modern language and multimodal AI has come on top of this one architecture, so understanding it underlies understanding the field.

For example, GPT, Claude, and Gemini are all transformer-based, processing each token by attending to every other token in their context.

Heard on the show

“And this doesn't contradict the theorems saying transformers can't exactly count: a coarse, revisable estimate of remaining length is a much more forgiving object than exact counting.”
Episode 204 — The Length Estimate Hiding Inside a Word-by-Word Model

Mentioned in 27 episodes

  1. 204
    The Length Estimate Hiding Inside a Word-by-Word Model
  2. 198
    The Model That Knows the Answer and Can't Say It
  3. 193
    Freeze Most of the Network: Where RL Improvement Actually Lives in a Transformer
  4. 182
    How a Tiny Model Too Weak to Plan Cuts a Bigger Agent's Hallucinations by 80%
  5. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  6. 158
    How Floating-Point Rounding Lets a Model Tell Which Chip It's On — And Misbehave
  7. 153
    Catching a Lie From the Inside, When the Words Look Completely Honest
  8. 145
    Building Forgetting Into a Language Model With One Extra Line of Code
  9. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
  10. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
  11. 094
    Chain-of-Thought Monitoring Fails Across Languages, and Worst Where It's Needed Most
  12. 091
    When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning
  13. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
  14. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
  15. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  16. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
  17. 053
    An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script
  18. 041
    When the Iteration Teaches the Model to Skip the Iteration
  19. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
  20. 038
    How LLMs Get Persuaded: One Attention Head, A Tetrahedron, And A Single Dial
  21. 036
    Sparse Attention Was the Wrong Frame. Treat It as Geometry Instead.
  22. 033
    Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval
  23. 032
    A Sticky-Note for Every Layer: Letting Transformers Remember What They Were Just Thinking
  24. 027
    When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure
  25. 023
    Why a Small Agent Confidently Overwrites Memories It Doesn't Understand
  26. 016
    Why Your Coding Agent Stalls While the GPU Runs Hot
  27. 002
    An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light

Related concepts

Attention Analysis Attention Heads Hybrid SSM/Attention In-Context Learning KV Cache Residual Stream Transformer Attention

Related terms

attention MLP layer residual