Definition
Plain language
A score for how surprised a model is by the next piece of data — lower means it predicted better.
As stated in the literature
Cross-entropy loss normalized per byte of raw input rather than per token, allowing comparison across tokenizers, modalities, and byte-encoded domains; equivalent to a compression rate.
Also called: BPB, bits per byte
Why it matters: Unlike per-token perplexity, bits-per-byte is comparable across tokenizers, making it the right metric when comparing very different model architectures.
For example, a model with 1.0 bits-per-byte on a text corpus is, on average, predicting each byte using one bit of information beyond random guessing.
Heard on the show
“… combo — and you have five minutes of wall-clock training on a single GPU to minimize validation bits-per-byte on a tokenized web corpus. …”Episode 053 — An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script