The second post-training phase is designed to generalize specialized reasoning capabilities to a broad range of scenarios and achieve comprehensive alignment.
Stage 3's core mechanism is to use rejection sampling to filter high-quality samples from the Stage 2 RL checkpoint and construct a large-scale training dataset. Reasoning data (roughly 600K samples) is evaluated via rule-based rewards or using DeepSeek-V3 as a generative reward model (LLM as Judge). Non-reasoning data (roughly 200K samples) reuses DeepSeek-V3's SFT dataset with enhanced diversity. These are combined and used for two rounds of supervised fine-tuning on DeepSeek-V3-Base, thereby transferring the reasoning capabilities that emerged on specialized tasks like math and programming to a broader range of general-purpose tasks, significantly improving output readability and general performance.
Stage 4 performs final alignment optimization on the Stage 3 SFT model using a combined reward strategy: for reasoning tasks (e.g., math, programming), it continues using rule-based accuracy rewards; for open-domain general tasks (e.g., creative writing, multi-turn dialogue), it introduces human-preference-based reward models to evaluate the helpfulness and harmlessness of outputs. Through this hybrid reward mechanism and training on diverse prompts covering all scenarios, the model maintains or even enhances its specialized reasoning capabilities while achieving comprehensive alignment with human values, ultimately producing the powerful yet practical final version of DeepSeek-R1.