Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

2026-04-27

AI Summary

This work addresses the limitation of existing large language models in multi-objective alignment, which typically rely on static preference weights and thus fail to capture the dynamic trade-offs among preferences during training. To overcome this, the authors propose Meta-Aligner, a framework that enables bidirectional co-optimization of preferences and policy through bilevel meta-learning. Specifically, a learnable preference weight network acts as a meta-learner, dynamically adjusting preference weights via a combination of rejection sampling and prompt-based adaptive weight generation to guide the alignment process. Experimental results demonstrate that Meta-Aligner significantly outperforms current state-of-the-art methods across multiple multi-objective alignment benchmarks, confirming the effectiveness and superiority of the proposed dynamic, bidirectional optimization paradigm.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Metareasoning and Metaheuristics
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

Problem

• Multi-Objective Alignment
• Preference Optimization
• Large Language Models
• Dynamic Preferences
• LLM Alignment

Innovation

• meta-learning
• bidirectional optimization
• dynamic preference
• multi-objective alignment
• preference-weight-net