DeepSeek-R1's training pipeline: the first post-training phase consists of two stages — SFT and RL.
Stage 1, Cold Start SFT, is designed to address the poor output readability and language mixing issues observed in pure RL training. By fine-tuning the base model on several thousand high-quality long chain-of-thought examples, it instills normalized reasoning patterns and language consistency, establishing a stable initial policy.
Building on this, Stage 2, Reasoning-Oriented RL, employs the GRPO algorithm with a combination of rule-based accuracy rewards and language consistency rewards. The model undergoes large-scale training on math, programming, and scientific reasoning tasks, significantly boosting reasoning capability while maintaining output readability.
These two stages work in synergy: SFT introduces human prior knowledge to normalize behavior, while RL allows the model to autonomously explore better reasoning strategies, ultimately achieving a balance between strong reasoning performance and good readability.
- Stage 1: Cold Start SFT
- Stage 2: Reasoning-Oriented RL
- Stage 3: Rejection Sampling & SFT
- Stage 4: RL for All Scenarios