Adaptive Preference Aggregation

2025-03-13

AI Summary

To address the lack of theoretical guarantees in RLHF for aggregating diverse, heterogeneous human preferences, this paper proposes a context-adaptive preference integration framework. Methodologically, it pioneers the coupling of Maximal Lotteries from social choice theory with a dynamic Pólya urn process, unifying Condorcet consistency with high-dimensional, non-stationary preference modeling. It further introduces Condorcet graph-structured analysis and an online adaptive preference weighting mechanism to enhance robustness against evolving user preferences. Empirically, the framework achieves a 27% improvement in Condorcet win rate and a 19% gain in NDCG@10 on recommendation and AI alignment benchmarks. The approach thus delivers both rigorous theoretical foundations—rooted in axiomatic social choice—and strong empirical performance, bridging a critical gap between preference aggregation theory and practical RLHF deployment.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

AI alignment, the challenge of ensuring AI systems act in accordance with human values, has emerged as a critical problem in the development of systems such as foundation models and recommender systems. Still, the current dominant approach, reinforcement learning with human feedback (RLHF) faces known theoretical limitations in aggregating diverse human preferences. Social choice theory provides a framework to aggregate preferences, but was not developed for the multidimensional applications typical of AI. Leveraging insights from a recently published urn process, this work introduces a preference aggregation strategy that adapts to the user's context and that inherits the good properties of the maximal lottery, a Condorcet-consistent solution concept.

Problem

• AI alignment with human values
• Limitations of RLHF in preference aggregation
• Adaptive strategy for multidimensional AI applications

Innovation

• Adaptive preference aggregation strategy
• Incorporates user context dynamically
• Utilizes maximal lottery properties