Definition
Plain language
The part of an AI that chops incoming text into the small pieces it actually reads.
As stated in the literature
Component that maps raw bytes or characters to model tokens (often via BPE); it operates on byte sequences without regard to whether a character is human-visible, which is why invisible characters still reach the model.
Also called: tokenizers
Why it matters: Because it processes raw characters without caring whether they're visible, invisible text still gets fed to the model just like normal words.
For example, the word 'unbelievable' might be split into smaller chunks like 'un', 'believ', and 'able' before the AI reads it.
Heard on the show
“Feed it to the model's tokenizer and nothing special happens — the tokenizer reads bytes, it doesn't ask "can a human see this?”Episode 208 — The Blank Space in Your AI Approval Box That Isn't Empty