Mutual-Taught for Co-adapting Policy and Reward Models

2025-05-17

AI Summary

To address co-degradation between policy models (PMs) and reward models (RM) in preference optimization—caused by distributional shift—this paper proposes a human-annotation-free mutual-teaching self-training framework. Methodologically, it introduces a novel bidirectional adaptive mechanism between PM and RM, and is the first to incorporate the Expectation-Maximization (EM) paradigm into preference alignment: in the E-step, the PM is updated using the current RM; in the M-step, the RM’s training data is dynamically relabeled using paired responses generated by the updated PM, enabling joint co-evolution. On AlpacaEval-2, the 8B policy model LLaMA-3-8B-Instruct-MT achieves a 54.1% length-controlled win rate; its counterpart 8B reward model, FsfairX-LLaMA3-RM-MT, matches the performance of GPT-4o-2024-08-06 on RewardBench.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, LLaMA-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1% on AlpacaEval-2, while our 8B reward model, FsfairX-LLaMA3-RM-MT, performs on par with GPT-4o-2024-08-06 on RewardBench.

Problem

• Addresses distribution shifts between policy and reward models
• Improves both models without human annotation
• Ensures reward model adapts to evolving policy distribution

Innovation

• Self-training method for co-adapting policy and reward models
• Iterative updates mirroring expectation-maximization algorithm
• Policy and reward models improve without human annotation