Preference Optimization with Multi-Sample Comparisons

2024-10-16

arXiv.org

AI Summary

Existing post-training methods for generative models—such as RLHF and DPO—rely on pairwise preference comparisons over single samples, limiting their ability to model population-level properties like diversity and bias. This work proposes the first preference optimization framework based on *multi-sample* comparisons, introducing two novel algorithms: mDPO and mIPO. These methods directly optimize collective characteristics of generated outputs at the set level, extending DPO and IPO with intra-group consistency constraints and noise-robust mechanisms. Experiments demonstrate that the proposed framework significantly outperforms single-sample baselines in enhancing output diversity, mitigating bias, and maintaining robustness under label noise. The results validate both the effectiveness and necessity of multi-sample comparison for modeling and optimizing population-level behavioral traits in generative models.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Recent advancements in generative models, particularly large language models (LLMs) and diffusion models, have been driven by extensive pretraining on large datasets followed by post-training. However, current post-training methods such as reinforcement learning from human feedback (RLHF) and direct alignment from preference methods (DAP) primarily utilize single-sample comparisons. These approaches often fail to capture critical characteristics such as generative diversity and bias, which are more accurately assessed through multiple samples. To address these limitations, we introduce a novel approach that extends post-training to include multi-sample comparisons. To achieve this, we propose Multi-sample Direct Preference Optimization (mDPO) and Multi-sample Identity Preference Optimization (mIPO). These methods improve traditional DAP methods by focusing on group-wise characteristics. Empirically, we demonstrate that multi-sample comparison is more effective in optimizing collective characteristics~(e.g., diversity and bias) for generative models than single-sample comparison. Additionally, our findings suggest that multi-sample comparisons provide a more robust optimization framework, particularly for dataset with label noise.

Problem

• Extends post-training with multi-sample comparisons for generative models
• Improves diversity and bias assessment in generative model outputs
• Provides robust optimization for datasets with label noise

Innovation

• Extends post-training with multi-sample comparisons
• Introduces mDPO and mIPO for group-wise optimization
• Enhances diversity and bias assessment in generative models