Definition
Plain language
An older, smaller language model often used as a quick text classifier.
As stated in the literature
A robustly optimized BERT-style encoder-only transformer, commonly fine-tuned as a classifier or used as a baseline in reasoning benchmarks.
Why it matters: Its compact size makes it a practical classifier and a familiar baseline against which newer models are measured.
For example, it can be fine-tuned to sort customer reviews into positive or negative without needing a full chat model.
Heard on the show
“A fine-tuned RoBERTa, trained on more than a million examples of a causal reasoning task, gets 35 percent on the hardest version of the test.”Episode 091 — When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning