The post-training pipeline is built on three pillars — reward modeling (RM), supervised fine-tuning (SFT), and direct preference optimization (DPO) — and is iterated over six rounds to progressively enhance model capabilities.
- First, to support complex features such as tool calling, a multi-message chat protocol is designed using special header and termination tokens;
- In each round, reward models covering different capability areas are trained on human-annotated preference data, and these reward models are used for subsequent rejection sampling;
- Next, the reward model is used to perform rejection sampling on prompts, and the resulting data, combined with synthetic data and other sources, forms the SFT data used to fine-tune the pretrained model (learning rate 10−5, with loss computed only on target responses);
- Building on this, direct preference optimization (DPO) is applied using the most recent batch of preference data, with two key improvements: masking formatting tokens to avoid conflicting learning objectives, and adding an NLL loss regularization term to stabilize training;
- Finally, models from different experiments at each stage are averaged to enhance robustness.