Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization

2026-06-25

AI Summary

Existing general-purpose role-playing agents struggle to maintain character consistency in out-of-distribution scenarios, primarily due to their reliance on behavioral imitation without human-like internal reasoning mechanisms. This work proposes Psy-CoT, a novel framework that introduces, for the first time, a psychology-inspired dynamic chain-of-thought reasoning structure, decomposing pre-role-play reasoning into three stages: interaction perception, psychological empathy, and logical construction. The framework further incorporates Role-Aware Policy Optimization (RAPO) for reinforcement learning training and innovatively designs a gradient weighting mechanism based on mutual information between character identity and output tokens to effectively suppress reward-model-induced deceptive expressions. Experiments demonstrate that Psy-CoT significantly outperforms existing chain-of-thought methods on CoSER, CharacterBench, and CharacterEval, with RAPO consistently surpassing GRPO across different model scales.

Abstract

Building general-purpose role-playing agents that faithfully portray any character from a natural-language profile remains challenging. The dominant paradigm -- supervised fine-tuning -- encourages behavioral mimicry without deep, human-like internal thought processes, resulting in poor out-of-distribution generalization. Therefore, we propose \textbf{Psy-CoT}, a psychology-grounded chain-of-thought framework that decomposes pre-response reasoning into three role-specific steps -- \emph{Interaction Perception}, \emph{Psychological Empathy}, and \emph{Logical Construction} -- so that the model \emph{thinks dynamically} from the profile rather than merely mimicking surface patterns. While structured reasoning provides a foundation, it alone is insufficient; reinforcement learning is essential to further align the model with character fidelity. However, we observe that under LLM-based reward models, both generic phrases that hack the reward model and genuinely role-specific phrases receive identical gradient signals -- this hacking accumulates over training, misleading the model into treating both as equally optimal choices. To address this, we propose \textbf{Role-Aware Policy Optimization (RAPO)}, which uses profile--token mutual information to weight gradients asymmetrically -- amplifying role-specific tokens under positive advantage while attenuating them under negative advantage. Experiments on CoSER, CharacterBench, and CharacterEval demonstrate that Psy-CoT outperforms existing role-playing CoT methods, and RAPO consistently surpasses GRPO across multiple model scales.

Problem

• role-playing agents
• out-of-distribution generalization
• reward hacking
• character fidelity
• behavioral mimicry

Innovation

• Psychology-Grounded Reasoning
• Chain-of-Thought
• Role-Aware Policy Optimization
• Mutual Information
• Reinforcement Learning