Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

2024-10-04

Conference on Empirical Methods in Natural Language Processing

AI Summary

Existing LLM preference alignment methods rely on binary preference labels, failing to capture fine-grained quality differences between model outputs. To address this, we propose Margin-BT—the first method integrating the Bradley-Terry model with an explicit, learnable quality margin to construct soft target probability distributions, optimized end-to-end via standard cross-entropy loss for both policy and reward modeling. This approach preserves training simplicity while substantially improving alignment accuracy and robustness. Experiments demonstrate consistent superiority over baselines on MT-bench and RewardBench; notably, a 7B model achieves state-of-the-art performance on RewardBench (as of June 2024) among models of comparable scale, with markedly reduced overfitting risk. The core innovation lies in the learnable margin mechanism, which endows preference supervision signals with enhanced discriminability and interpretable physical meaning—quantifying minimal perceptible quality differences between outputs.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date. Despite their success, existing methods typically rely on simple binary labels, such as those indicating preferred outputs in pairwise preferences, which fail to capture the subtle differences in relative quality between pairs. To address this limitation, we introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models. Specifically, given quality margins in pairwise preferences, we design soft target probabilities based on the Bradley-Terry model, which are then used to train models with the standard cross-entropy objective. Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench. Notably, the 7B model trained with MMPO achieves state-of-the-art performance on RewardBench as of June 2024, outperforming other models of the same scale. Our analysis also shows that MMPO is more robust to overfitting, leading to better-calibrated models.

Problem

• Improves model alignment using granular feedback margins
• Addresses limitations of binary labels in preference optimization
• Enhances robustness and calibration in LLM training

Innovation

• Incorporates relative quality margins into optimization
• Uses soft target probabilities from Bradley-Terry model
• Trains models with standard cross-entropy objective