A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

2025-05-21

AI Summary

This work investigates the theoretical impact of noisy labels in offline alignment under the coupled effects of local differential privacy (LDP) and adversarial label corruption. We consider two sequential scenarios: “Label-then-Corrupt” (LTC) and “Corrupt-then-Label” (CTL). Under a linear reward modeling assumption, we develop a reduction framework that recasts RLHF and DPO optimization as noisy logistic regression parameter estimation, integrating tools from differential privacy and robust statistics. We establish, for the first time, a strict optimality separation: LTC is provably harder to optimize than CTL, yielding a formal theoretical gap in their achievable performance. Departing from prior single-factor analyses—focusing exclusively on either privacy or corruption—we propose the first unified theoretical framework characterizing offline alignment capability under *multiple concurrent noise sources*. This advances the foundational understanding of safe, robust preference alignment and significantly extends the theoretical frontier of alignment under security constraints.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

In this paper, we theoretically investigate the effects of noisy labels in offline alignment, with a focus on the interplay between privacy and robustness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under different privacy-corruption scenarios, such as Local differential privacy-then-Corruption (LTC), where human preference labels are privatized before being corrupted by an adversary, and Corruption-then-Local differential privacy (CTL), where labels are corrupted before privacy protection. Our analysis leverages a reduction framework that reduces the offline alignment problem under linear modeling assumptions to parameter estimation in logistic regression. This framework allows us to establish an interesting separation result between LTC and CTL, demonstrating that LTC presents a greater challenge than CTL in offline alignment, even under linear models. As important by-products, our findings also advance the state-of-the-art theoretical results in offline alignment under privacy-only or corruption-only scenarios.

Problem

• Analyzing noisy labels' impact on offline alignment privacy and robustness
• Comparing RLHF and DPO under privacy-corruption scenarios LTC and CTL
• Establishing LTC as more challenging than CTL in offline alignment

Innovation

• Unified analysis of RLHF and DPO under privacy-corruption
• Reduction framework for offline alignment to logistic regression
• Separation result between LTC and CTL scenarios