Unbiased Alignment for Large Language Models with Noisy Preferences

2026-07-03

AI Summary

This work addresses the susceptibility of existing large language model alignment methods to noise in human preference data, which often induces training bias. To mitigate this issue, the authors propose a theoretically grounded unbiased alignment framework that introduces, for the first time, an Unbiased Reward Model (URM) loss and an Unbiased Direct Preference Optimization (UDPO) loss. These loss functions enable unbiased learning directly from noisy preference data without requiring clean labels. The proposed approach exhibits strong noise tolerance, parameter backward compatibility, and classification calibration. Extensive experiments demonstrate that the method significantly outperforms current state-of-the-art techniques across multiple benchmark datasets, thereby validating both its theoretical soundness and practical robustness.

Abstract

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.

Problem

• unbiased alignment
• noisy preferences
• large language models
• preference noise
• human feedback

Innovation

• Unbiased Alignment
• Noisy Preferences
• Direct Preference Optimization
• Reward Modeling
• Noise-Tolerant Learning