Llama 2's RLHF stage is the core process that further optimizes the model's helpfulness and safety after SFT produces an initial dialogue model. It follows a three-step pipeline: first, based on human preference data comparing multiple responses, two separate reward models are trained — a helpfulness reward model for evaluating response quality and a safety reward model for judging content risk. Then, using these two reward models as "judges," the model undergoes multiple rounds of iterative fine-tuning via reinforcement learning algorithms such as proximal policy optimization (PPO) and rejection sampling. PPO continuously optimizes the policy to pursue higher rewards, while rejection sampling selects the best candidate from multiple generated responses. This stage also integrates red teaming to generate adversarial prompts that improve model robustness, and employs safety context distillation to internalize safety guidelines into the model. Ultimately, this refined RLHF pipeline enables Llama 2-Chat to provide highly helpful responses in human evaluations while significantly reducing harmful outputs, establishing it as an open-source conversational AI approaching the level of closed-source models.
We have seen RLHF in the InstructGPT topic; it would go on to become a standard component of chat models.