Definition
Iterative refinement has a model improve its output across multiple passes, often using its own critique or an external verifier as feedback. It often helps; it sometimes hurts (the model can talk itself out of correct answers); the wins are tied to whether the critique signal is reliable.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- Self-Refine: Iterative Refinement with Self-Feedback
- Universal Transformers
- Deep Equilibrium Models
- Looped Transformers as Programmable Computers
- FunSearch: Making new discoveries in mathematics using large language models
- OpenEvolve: Open-Source Implementation of AlphaEvolve
- FunSearch: Making new discoveries in mathematical sciences using large language models
- FunSearch: Making New Discoveries in Mathematical Sciences Using Large Language Models
- Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs
- Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters
- TextGrad: Automatic 'Differentiation' via Text
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
- Let's Verify Step by Step
- Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
- Training Large Language Models to Reason in a Continuous Latent Space
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- AIDE: AI-Driven Exploration in Machine Learning Research
- Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
- Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs