Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

2025-02-24

AI Summary

Existing RLHF methods rely on the Bradley–Terry (BT) model, which imposes restrictive structural assumptions and limits expressiveness in capturing complex human preferences. Method: We propose a general preference alignment framework that does not assume BT structure, formulating preference learning as a two-player zero-sum game and approximating the Nash equilibrium strategy via optimistic online mirror descent (Optimistic OMD). Contribution/Results: Our approach is the first to eliminate the BT assumption entirely in LLM alignment. Theoretically, it achieves an $O(T^{-1})$ convergence rate for the duality gap—strictly faster than the $O(T^{-1/2})$ rate of conventional RLHF. Empirically, it significantly outperforms state-of-the-art RLHF methods across multiple mainstream benchmarks, demonstrating that general preference modeling substantially enhances generalization and alignment capability.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $O(T^{-1})$ bound on the duality gap, improving upon the previous $O(T^{-1/2})$ result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.

Problem

• Aligning LLMs with human preferences
• General preferences without BT model
• Optimistic online mirror descent integration

Innovation

• Optimistic Online Mirror Descent
• General Preference Alignment
• Nash Policy Approximation