Definition
Scalable oversight is the research program of supervising AI systems whose outputs we can’t fully evaluate ourselves — because the model is more capable than the human, or the domain is too complex. Debate, recursive reward modeling, and constitutional AI are all proposed answers.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- Alignment faking in large language models
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions
- OpenAI o1 System Card
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisors
- Constitutional AI: Harmlessness from AI Feedback
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Auditing Language Models for Hidden Objectives
- Scalable AI Safety via Doubly-Efficient Debate
- AI Safety via Debate
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- AI Control: Improving Safety Despite Intentional Subversion
- Sabotage Evaluations for Frontier Models
- LeanDojo: Theorem Proving with Retrieval-Augmented Language Models