The SFT (Supervised Fine-Tuning) phase of Gemma-1 is the critical bridge connecting its powerful pretrained base model to the final, usable instruction model. At its core, it calibrates the model using high-quality instruction data. This phase employs an innovative, model-based automatic evaluation approach to data construction: first, human-generated instruction data is mixed with data synthesized by larger models; second, a "LM side-by-side evaluation" pipeline filters the data — different candidate models generate responses to the same prompt, and a stronger "judge" model evaluates their quality, automatically constructing high-quality training pairs focused on instruction following, factuality, creativity, and safety.
During training, all conversation data is formatted with special tokens that explicitly mark conversation roles and turn boundaries. This carefully designed SFT process not only significantly improves the model's ability to follow complex instructions but also lays a solid foundation for preference alignment in the subsequent Reinforcement Learning from Human Feedback (RLHF) phase.