Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment

2025-08-11

AI Summary

Supervised fine-tuning (SFT) is constrained by fixed offline data distributions, while reinforcement learning (RL) suffers from low sample efficiency and strong dependence on high-quality base models. To unify the strengths of both paradigms, this paper proposes GRAO—a novel alignment framework. Its core innovations are: (1) multi-sample generation coupled with a group-wise direct alignment loss, enabling fine-grained preference modeling via intra-group relative ranking; and (2) a reference-aware parameter update mechanism that jointly incorporates reward signals and supervised gradients, enhancing optimization stability and model evolvability. Evaluated across diverse alignment benchmarks, GRAO outperforms SFT, DPO, PPO, and GRPO by 57.70%, 17.65%, 7.95%, and 5.18%, respectively—demonstrating substantial gains in both alignment fidelity and sample efficiency.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Alignment methodologies have emerged as a critical pathway for enhancing language model alignment capabilities. While SFT (supervised fine-tuning) accelerates convergence through direct token-level loss intervention, its efficacy is constrained by offline policy trajectory. In contrast, RL(reinforcement learning) facilitates exploratory policy optimization, but suffers from low sample efficiency and stringent dependency on high-quality base models. To address these dual challenges, we propose GRAO (Group Relative Alignment Optimization), a unified framework that synergizes the respective strengths of SFT and RL through three key innovations: 1) A multi-sample generation strategy enabling comparative quality assessment via reward feedback; 2) A novel Group Direct Alignment Loss formulation leveraging intra-group relative advantage weighting; 3) Reference-aware parameter updates guided by pairwise preference dynamics. Our theoretical analysis establishes GRAO's convergence guarantees and sample efficiency advantages over conventional approaches. Comprehensive evaluations across complex human alignment tasks demonstrate GRAO's superior performance, achieving 57.70%,17.65% 7.95% and 5.18% relative improvements over SFT, DPO, PPO and GRPO baselines respectively. This work provides both a theoretically grounded alignment framework and empirical evidence for efficient capability evolution in language models.

Problem

• Enhancing language model alignment via unified SFT and RL integration
• Improving sample efficiency in alignment with multi-sample generation
• Optimizing policy updates using group relative advantage weighting

Innovation

• Multi-sample generation for reward feedback
• Group Direct Alignment Loss weighting
• Reference-aware parameter updates dynamics