Multi-Task Reward Learning from Human Ratings

2025-06-10

AI Summary

Existing RLHF approaches model human decision-making as a single classification or regression task, failing to capture its inherent multi-strategy nature and uncertainty. Method: We propose a multitask reward learning framework that jointly models preference classification (pairwise choice) and score regression (scalar rating prediction), enabling robust implicit reward function inference without explicit reward signals. To address task heterogeneity and uncertainty, we introduce a learnable dynamic weighting mechanism that explicitly models inter-task uncertainty and adaptively balances their contributions. Furthermore, we incorporate differentiable weight scheduling and synthetic score simulation to enhance training stability and generalization. Results: Extensive experiments demonstrate that our method significantly outperforms mainstream score-based RLHF baselines; notably, in several scenarios, it even surpasses conventional reinforcement learning approaches, establishing new state-of-the-art performance in reward modeling and policy optimization.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Reinforcement learning from human feeback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.

Problem

• Modeling multi-task human decision-making in RLHF
• Balancing classification and regression for reward inference
• Improving RL performance using adaptive human rating strategies

Innovation

• Multi-task RLHF with human ratings
• Learnable weights balance classification and regression
• Adaptive strategy emphasis in reward learning