UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learning

2026-01-14

AI Summary

This work addresses the limitations of existing user simulators, which rely on static, context-agnostic user profiles that struggle to generalize to novel scenarios and are vulnerable to manipulation by agents due to their lack of strategic reasoning. To overcome these challenges, we propose a novel user profiling framework that integrates static personas with dynamic goals, enabling goal-driven, strategic responses through a dedicated reasoning mechanism. We further enhance the decision-making capabilities of the user language model via supervised fine-tuning and multi-reward reinforcement learning. As the first approach to incorporate dynamic goals and multi-reward reinforcement learning into user simulation, our method significantly outperforms current state-of-the-art techniques across multiple benchmarks, demonstrating exceptional cross-scenario generalization and robustness against adversarial manipulation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

User simulators serve as the critical interactive environment for agent post-training, and an ideal user simulator generalizes across domains and proactively engages in negotiation by challenging or bargaining. However, current methods exhibit two issues. They rely on static and context-unaware profiles, necessitating extensive manual redesign for new scenarios, thus limiting generalizability. Moreover, they neglect human strategic thinking, leading to vulnerability to agent manipulation. To address these issues, we propose UserLM-R1, a novel user language model with reasoning capability. Specifically, we first construct comprehensive user profiles with both static roles and dynamic scenario-specific goals for adaptation to diverse scenarios. Then, we propose a goal-driven decision-making policy to generate high-quality rationales before producing responses, and further refine the reasoning and improve strategic capabilities with supervised fine-tuning and multi-reward reinforcement learning. Extensive experimental results demonstrate that UserLM-R1 outperforms competitive baselines, particularly on the more challenging adversarial set.

Problem

• user simulator
• generalizability
• strategic reasoning
• human-like negotiation
• context-awareness

Innovation

• User Language Model
• Multi-Reward Reinforcement Learning
• Goal-Driven Reasoning
• Dynamic User Profiling
• Strategic Interaction