ChatGLM's alignment process is primarily achieved through multi-stage, combined alignment techniques. First, supervised fine-tuning (SFT) uses high-quality real human instruction-output paired data to awaken the model's instruction-following capability, laying the foundation for subsequent optimization. Next, reinforcement learning from human feedback (RLHF, specifically implemented through the ChatGLM-RLHF pipeline) performs fine-grained calibration: by collecting human preference data across multiple responses, training a reward model, and using algorithms such as PPO/DPO to optimize the policy model, the model's outputs become better aligned with human subjective preferences across dimensions such as safety, factuality, relevance, and helpfulness. This effectively mitigates issues such as inappropriate response refusals, disfluent bilingual mixed generation, and incoherent multi-turn dialogues.
To reduce dependence on costly human feedback data, Zhipu AI also developed the Self-Contrast technique, a feedback-free alignment strategy that leverages the target large model itself to automatically generate large quantities of negative samples for constructing contrastive data to efficiently train the reward model.