WildReward: Learning Reward Models from In-the-Wild Human Interactions

2026-02-09

AI Summary

This work proposes WildReward, a novel reward modeling approach that eliminates the need for costly and hard-to-scale human preference annotations by leveraging natural user interactions with large language models, such as those in the WildChat dataset. By analyzing interaction logs, filtering noise, and extracting implicit feedback, the authors construct a high-quality training set of 186k samples and train the reward model using ordinal regression—without requiring explicit preference pairs. The method demonstrates that user diversity positively contributes to model performance. WildReward matches or surpasses conventional reward models in effectiveness while exhibiting superior calibration and cross-sample consistency, and it significantly enhances multi-task performance when integrated into online Direct Preference Optimization (DPO) training.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.

Problem

• reward models
• in-the-wild interactions
• human feedback
• large language models
• preference pairs

Innovation

• reward modeling
• in-the-wild interactions
• ordinal regression
• implicit feedback
• online DPO