In its RLHF phase, Gemma-1 does not use the industry-common PPO + RM combination. Instead, it employs a variant of the REINFORCE algorithm with a Kullback–Leibler (KL) divergence regularization term.
Specifically, the process is as follows: first, under the Bradley-Terry model framework, a reward model (RM) is trained using human-annotated preference pair data. Then, an improved REINFORCE algorithm (rather than PPO) is used to optimize the policy model (i.e., the Gemma model). The objective is to maximize the reward assigned by the reward model while constraining the policy model through a KL divergence penalty term, preventing its output distribution from deviating excessively from the initial SFT model. This design aims to effectively align with human preferences while maintaining model stability and mitigating reward hacking.