Definition
Plain language
The step that turns the model's numbered pieces back into readable text.
As stated in the literature
The inverse of the tokenizer: maps generated token IDs back to a byte/character string, handling byte-level merges, whitespace conventions, and Unicode normalization — implementations differ subtly across engines.
Also called: detokenization, detokenizers
Why it matters: Without it you would see numbers instead of sentences, and small disagreements between detokenizers about spacing or accented characters are enough to tell two otherwise identical systems apart.
For example, the detokenizer turns the model's output pieces "un", "believ", "able" back into the single word "unbelievable", with the right spaces around it.
Heard on the show
“In both cases, the text the model just generated makes a round trip, back through the engine's detokenizer and its templater, and lands in front of the model again as input.”Episode 272 — How a Model Guesses Which Engine Is Running It, From a Wrong Date