Definition
Plain language
The basic unit of text a language model reads or writes — roughly a word or part of a word.
As stated in the literature
A discrete unit from a tokenizer's vocabulary, often a subword piece, that language models consume and produce one at a time.
Also called: tokens
Why it matters: Tokenization shapes context length limits, billing, and even how well models handle numbers, code, and non-English languages.
For example, the word 'unbelievable' might be split into the tokens 'un', 'believ', and 'able' before the model sees it.
Heard on the show
“… The numbers, in commute-friendly form: about a hundred and forty-six million tokens of reasoning, around three thousand calls to its underlying language model, twelve hundred tool calls …”Episode 002 — An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light
Related concepts
AI Text Detection
Attention Analysis
Capability vs. Efficiency
Chain of Thought
Denial-of-Wallet
Entropy Gating
Greedy Coordinate Gradient
KV Cache
Latent Diffusion
Logit Lens
Long Context
Loss Aggregation
Residual Stream
Side-Channel Attack
Speculative Decoding
Token-Level Analysis
Transformer Attention
Variational Autoencoder
Watermarking