The post-training stage of Gemma 2 aims to transform the general-purpose pretrained foundation model into a safe, helpful, and instruction-following conversational model. The core pipeline consists of Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).
In the SFT stage, the model is trained on a purely English instruction-response pair dataset consisting of a mix of human-generated and more powerful "teacher model" synthesized content. A data-driven "side-by-side selection" method is employed to automatically filter high-quality data that specifically targets the model's weaknesses, while the data undergoes rigorous filtering to remove unsafe content. Subsequently, in the RLHF stage, the model is first optimized under the guidance of a much larger, standalone reward model (focused on evaluating conversational capability), which is trained on human preference data. The model's behavior is then fine-tuned via a policy optimization algorithm (such as a PPO variant) with a KL divergence constraint, aligning its outputs more closely with human preferences for helpfulness and harmlessness while preventing excessive deviation from the SFT foundation. Finally, model merging techniques (such as WARP) are applied to average multiple policy models, improving robustness and stability.