Definition
Plain language
A model design that runs the same layers over and over to think harder, instead of stacking more layers.
As stated in the literature
An architecture that applies a shared block of transformer layers iteratively during a forward pass, trading sequential compute for reasoning depth at fixed parameter count; Parcae and Universal Transformers are examples.
Also called: looped transformers, looped Transformer
Why it matters: It lets a model think harder by spending more steps rather than more parameters, trading time for reasoning depth.
For example, instead of stacking more layers, the model runs the same set of layers several times in a row to reason through a hard problem.
Heard on the show
“Now, fixed-loop training — the looped Transformer way — has no such fence.”Episode 041 — When the Iteration Teaches the Model to Skip the Iteration