User-centric Subjective Leaderboard by Customizable Reward Modeling

2025-08-12

AI Summary

Existing LLM benchmarks predominantly focus on verifiable tasks and neglect modeling user-subjective preferences, hindering personalized model selection. Method: We propose the first user-centric subjective leaderboard (USL), powered by a customized reward model (CRM) that enables dynamic, configurable preference evaluation. The CRM—parameterized with only 4B parameters—is trained on over 10,000 real-world human preference annotations and introduces two key innovations: dynamic preference modeling and a negative-correlation conflict suppression mechanism to capture preference diversity and intrinsic contradictions. Contribution/Results: Experiments demonstrate that CRM significantly outperforms state-of-the-art models—including GPT-4.1 and Gemini-2.5-Pro—across multiple topics and evaluation criteria. USL accurately reflects individual preference differences and substantially improves the alignment and practical utility of model selection in real-world applications.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Existing benchmarks for large language models (LLMs) predominantely focus on assessing their capabilities through verifiable tasks. Such objective and static benchmarks offer limited utility for practical LLM selection, making it difficult for users to find suitable models for their individual needs. To bridge this gap, we present the first User-Centric Subjective Leaderboard (USL), which provides a preference-driven, dynamic ranking of LLMs across diverse real-world scenarios. Our work is built upon a thorough investigation of real human preference data, involving more than 10K subjective queries. Our investigation reveals significant diversity and contradictions in human preferences, which limit the effectiveness of state-of-the-art reward models. To address this, we introduce Customizable Reward Models (CRMs). With only 4B parameters, our CRM surpasses the performance of leading models such as GPT-4.1 and Gemini-2.5-pro, showing exceptional generalization capabilities across new topics and criteria. The USL, powered by CRMs, exhibits strong negative correlations to contradictory preferences.

Problem

• Assessing LLMs with verifiable tasks lacks practical user utility
• Existing benchmarks fail to address diverse human preferences effectively
• Static leaderboards struggle with dynamic real-world user needs

Innovation

• User-Centric Subjective Leaderboard for dynamic LLM ranking
• Customizable Reward Models with 4B parameters
• CRM outperforms GPT-4.1 and Gemini-2.5-pro