Definition
Plain language
A rulebook that randomly builds nested, sentence-like structures.
As stated in the literature
Generative grammar with production probabilities, yielding hierarchical sequences; used as a synthetic pretraining source producing language-shaped structure.
Also called: probabilistic context-free grammars, PCFG, PCFGs
Why it matters: It can manufacture unlimited language-shaped training data with known structure, letting researchers study what models learn from structure alone, separate from meaning.
For example, a rulebook might say a sentence is a noun phrase plus a verb phrase, and each of those can itself expand further, producing nested text that looks structured without meaning anything.
Heard on the show
“It generates sequences from probabilistic context-free grammars, which produce hierarchical, language-shaped structure.”Episode 280 — Two Random Networks Teach Each Other To Predict Real Data