AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment

2025-11-12

AI Summary

Current language model preference optimization suffers from an “overfitting-underfitting dilemma”: fixed margins induce gradient redundancy on correctly ranked pairs (overfitting) while providing insufficient correction for incorrectly ranked pairs (underfitting). To address this, we propose AMaPO—Adaptive Margin Preference Optimization—which formally characterizes this dilemma and introduces the first instance-level adaptive margin mechanism. Leveraging Z-score normalization and exponential scaling, AMaPO dynamically adjusts the margin in the Bradley–Terry loss to intelligently redistribute gradients. Integrated into offline preference optimization frameworks, it requires no additional annotations or online interaction. On standard benchmarks, AMaPO consistently improves ranking accuracy and downstream task alignment. Ablation studies and empirical analysis confirm that it effectively alleviates gradient allocation imbalance, enhancing both training stability and generalization performance.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models. However, their effectiveness is critically dependent on ranking accuracy, a metric where further gains are highly impactful. This limitation arises from a fundamental problem that we identify and formalize as the Overfitting-Underfitting Dilemma: current margin designs cause models to apply excessive, wasteful gradients to correctly ranked samples (overfitting) while providing insufficient corrective signals for misranked ones (underfitting). To resolve this dilemma, we propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled algorithm. AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones. Extensive experiments on widely used benchmarks demonstrate that AMaPO not only achieves better ranking accuracy and superior downstream alignment performance, but targeted analysis also confirms that it successfully mitigates the core overfitting and underfitting issues.

Problem

• Current preference optimization methods suffer from overfitting-underfitting dilemma in ranking
• Fixed margin designs cause wasteful gradients on correct samples and insufficient correction on misranked ones
• Existing approaches critically depend on ranking accuracy with limited improvement gains

Innovation

• Adaptive margin dynamically reallocates learning effort
• Z-normalization and exponential scaling refine margins
• Amplifies gradients for misranked samples while suppressing correct ones