The Differences Between Direct Alignment Algorithms are a Blur

2025-02-03

AI Summary

Prior work attributes performance disparities among direct alignment algorithms (DAAs) to reward modeling or loss-function specifics, obscuring the true source of variation. Method: This study identifies the optimization objective’s form—pairwise versus pointwise—as the fundamental determinant of DAA efficacy, and proposes a unified framework featuring an explicit supervised fine-tuning (SFT) stage and a tunable β parameter, enabling single-stage methods (enhanced ORPO/ASFT) to match two-stage approaches (e.g., DPO) in performance. Contribution/Results: Empirical analysis confirms that the pairwise objective—not implicit reward design—is decisive for alignment quality, correcting a longstanding misconception in the field. On AlpacaEval 2, enhanced ORPO and ASFT achieve gains of +3.46 and +8.27 points, respectively, reaching DPO-level performance. Furthermore, the study establishes a new evaluation benchmark centered on objective-form classification, advancing principled DAA comparison and design.

Technology Category

• Search and Optimization: Algorithm Configuration
• Machine Learning: Optimization
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Direct Alignment Algorithms (DAAs) simplify language model alignment by replacing reinforcement learning (RL) and reward modeling (RM) in Reinforcement Learning from Human Feedback (RLHF) with direct policy optimization. DAAs can be classified by their ranking losses (pairwise vs. pointwise), by the rewards used in those losses (e.g., likelihood ratios of policy and reference policy, or odds ratios), or by whether a Supervised Fine-Tuning (SFT) phase is required (two-stage vs. one-stage). We first show that one-stage methods underperform two-stage methods. To address this, we incorporate an explicit SFT phase and introduce the $eta$ parameter, controlling the strength of preference optimization, into single-stage ORPO and ASFT. These modifications improve their performance in Alpaca Eval 2 by +$3.46$ (ORPO) and +$8.27$ (ASFT), matching two-stage methods like DPO. Further analysis reveals that the key factor is whether the approach uses pairwise or pointwise objectives, rather than the specific implicit reward or loss function. These results highlight the importance of careful evaluation to avoid premature claims of performance gains or overall superiority in alignment algorithms.

Problem

• Direct Alignment Algorithms
• Language Model Optimization
• Evaluation Criteria

Innovation

• Direct Alignment Algorithm Enhancement
• Fine-tuning Learning Phase
• Beta Parameter Optimization