Multi-Objective Exploration and Preference Optimization via Mutual Information

2026-07-01

AI Summary

This work addresses the challenge in multi-objective alignment where uncertainty-driven exploration often leads to insufficient alignment between generated responses and preference vectors, as well as overlapping reward distributions. To resolve this, the paper introduces the MI-EPO framework, which for the first time incorporates mutual information theory into multi-objective language model alignment. By maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors, MI-EPO unifies exploration and alignment into a cohesive process. A probabilistic routing mechanism is further designed to decouple objective-specific alignment from preference-aware exploration. Integrated with online direct preference optimization, the proposed method significantly enhances consistency between model outputs and target preference vectors, improves controllability, and achieves stable trade-offs across multiple objectives in both safety alignment and assistant capability tasks.

Abstract

Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.

Problem

• multi-objective alignment
• preference optimization
• large language models
• preference vectors
• reward distribution overlap

Innovation

• mutual information
• multi-objective alignment
• preference optimization
• probabilistic routing
• large language models