GroupDPO: Memory efficient Group-wise Direct Preference Optimization

2026-04-16

AI Summary

This work addresses the limitations of existing preference optimization methods, which typically rely on a single positive–negative response pair per prompt and thus overlook the rich supervisory signals present in multiple candidate responses, while group-wise approaches suffer from prohibitive memory costs. To overcome this, the authors propose a memory-efficient group-based direct preference optimization method that decouples samples during backpropagation to reduce peak memory consumption without discarding gradient information, thereby enabling larger group sizes during training. Additionally, they introduce a negative log-likelihood regularization term on positive samples, which substantially enhances both performance and training stability. Experimental results demonstrate that the proposed approach consistently outperforms conventional pairwise training under both offline and online alignment settings, confirming its effectiveness and scalability.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we introduce a memory-efficient group-wise preference optimization algorithm that preserves gradients while decoupling samples during backpropagation, substantially reducing peak memory usage, which enables scalable training with larger group sizes. Across both offline and online alignment settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.

Problem

• preference optimization
• group-wise learning
• memory efficiency
• large language models
• preference datasets

Innovation

• Group-wise Preference Optimization
• Memory Efficiency
• Direct Preference Optimization
• Large Language Models
• Negative Log-Likelihood