🤖 AI Summary
Large language models (LLMs) often internalize societal biases, hindering value alignment with human preferences. This paper addresses two key limitations of existing preference optimization methods: distributional shift in RLHF and insufficient robustness of DPO. We propose a two-stage hybrid preference optimization framework: first, preference data are stratified into “easy” and “hard” samples based on reward-gap thresholds; second, an initial policy is trained via DPO on the easy subset, then fine-tuned online via PPO-RLHF on the hard subset—using the DPO-trained policy as a dynamic reference model. To our knowledge, this is the first work to leverage DPO-trained policies as reference models in RLHF, establishing a synergistic paradigm that balances training efficiency and policy robustness. Extensive experiments on HH-RLHF and TLDR demonstrate significant improvements over state-of-the-art baselines. Both GPT-4-based automated evaluation and human assessment confirm that our method yields safer, more human-preferred outputs.
📝 Abstract
Large Language Models (LLMs) have become increasingly popular due to their ability to process and generate natural language. However, as they are trained on massive datasets of text, LLMs can inherit harmful biases and produce outputs that are not aligned with human values. This paper studies two main approaches to LLM alignment: Reinforcement Learning with Human Feedback (RLHF) and contrastive learning-based methods like Direct Preference Optimization (DPO). By analyzing the stability and robustness of RLHF and DPO, we propose MPO (Mixed Preference Optimization), a novel method that mitigates the weaknesses of both approaches. Specifically, we propose a two-stage training procedure: first train DPO on an easy dataset, and then perform RLHF on a difficult set with DPO model being the reference model. Here, the easy and difficult sets are constructed by a well-trained reward model that splits response pairs into those with large gaps of reward (easy), and those with small gaps (difficult). The first stage allows us to obtain a relatively optimal policy (LLM) model quickly, whereas the second stage refines LLM with online RLHF, thus mitigating the distribution shift issue associated with DPO. Experiments are conducted on two public alignment datasets, namely HH-RLHF and TLDR, demonstrating the effectiveness of MPO, both in terms of GPT4 and human evaluation.