Definition
Plain language
An early, much smaller language model from OpenAI, now mostly used as a historical comparison point.
As stated in the literature
OpenAI's 2019 transformer language model, with variants up to 1.5B parameters; used as a low baseline in architecture comparisons, where beating it is necessary but unimpressive for a modern model.
Also called: GPT-two
Why it matters: It serves as a low historical baseline, useful for sanity checks even though surpassing it no longer signals real strength.
For example, researchers sometimes compare a new model against GPT-2 to confirm it clears a basic bar, since simply beating it is expected nowadays.
Heard on the show
“The authors also throw in a comparison to GPT-two at three-hundred-forty-five million parameters.”Episode 033 — Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval