Intrinsic Mutual Information as a Modulator for Preference Optimization

2026-04-27

AI Summary

This work addresses the inefficiency of offline preference optimization methods—such as Direct Preference Optimization (DPO)—which often require extensive hyperparameter tuning. To mitigate this issue, the authors propose RMiPO, a novel framework that introduces response-level intrinsic mutual information as a dynamic modulation mechanism for preference signals. This approach effectively decouples the contribution of each sample pair to preference learning with negligible computational overhead. By integrating concepts from offline reinforcement learning and information theory, RMiPO substantially reduces sensitivity to hyperparameters. Experimental results demonstrate that RMiPO consistently outperforms existing methods across multiple tasks while reducing training costs by more than 15%.

Abstract

Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15\%. Our code is available at https://github.com/liavonpenn/rmipo.

Problem

• Offline Preference Optimization
• Hyperparameter Tuning
• Large Language Models
• Mutual Information
• Preference Alignment

Innovation

• Intrinsic Mutual Information
• Preference Optimization
• Hyperparameter Modulation
• Offline RLHF
• Response-level Decoupling