A Shared Low-Rank Adaptation Approach to Personalized RLHF

2025-03-24

AI Summary

Existing RLHF frameworks assume homogeneous human preferences and rely on a single, shared reward model, thereby neglecting individual heterogeneity and hindering personalized alignment and user trust. To address this, we propose the first personalized RLHF framework based on shared low-rank adaptation (LoRA), which jointly models cross-user common structures and user-specific variations in parameter space—without imposing strong sharing assumptions. Our method enables efficient learning of personalized reward models from limited local data and provides a theoretically grounded upper bound on sample complexity. Extensive experiments on real-world datasets demonstrate that our approach significantly improves generalization under few-shot settings, enhances personalized alignment accuracy, and increases user satisfaction compared to baseline methods.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively homogeneous and can be captured by a single, unified reward model. This assumption overlooks the inherent diversity and heterogeneity across individuals, limiting the adaptability of RLHF to personalized scenarios and risking misalignments that can diminish user satisfaction and trust in AI systems. In this paper, we address these challenges by introducing Low-Rank Adaptation (LoRA) into the personalized RLHF framework. We apply LoRA in the the aggregated parameter space of all personalized reward functions, thereby enabling efficient learning of personalized reward models from potentially limited local datasets. Our approach exploits potential shared structures among the local ground-truth reward models while allowing for individual adaptation, without relying on restrictive assumptions about shared representations as in prior works. We further establish sample complexity guarantees for our method. Theoretical analysis demonstrates the effectiveness of the proposed approach in capturing both shared and individual-specific structures within heterogeneous human preferences, addressing the dual challenge of personalization requirements and practical data constraints. Experimental results on real-world datasets corroborate the efficiency of our algorithm in the personalized RLHF setting.

Problem

• Diverse human preferences challenge unified RLHF reward models
• Personalized RLHF needs efficient adaptation to individual datasets
• Balancing shared and unique preference structures in RLHF

Innovation

• Low-Rank Adaptation for personalized RLHF
• Efficient learning from limited local datasets
• Captures shared and individual-specific preference structures