Definition
SFT (Supervised Fine-Tuning) trains a pretrained model on (input, target output) pairs to teach a specific behavior or format. It’s the simplest post-training method and the first step in most modern alignment pipelines before any RL.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- LUFFY: Learning to Reason Under Off-Policy Guidance
- Fine-tuning aligned language models compromises safety, even when users are not the ones fine-tuning
- Training language models to follow instructions with human feedback
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- DAgger: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
- STaR: Bootstrapping Reasoning With Reasoning
- Toolformer: Language Models Can Teach Themselves to Use Tools