One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence

2026-02-03

AI Summary

This work addresses the challenge of enabling AI to autonomously acquire complex social intelligence—such as empathy, persuasion, and compromise—in multi-turn group conversations, moving beyond static or single-turn optimization paradigms. The authors propose OMAR, a novel framework that, for the first time, allows a single model to simultaneously embody all roles in multi-agent, multi-turn dialogues through self-play reinforcement learning, thereby learning long-term objectives and social norms via dynamic interaction. OMAR introduces a hierarchical advantage estimation mechanism that integrates turn-level and token-level advantage computations, significantly enhancing training stability in extended dialogues. Experiments demonstrate that the model spontaneously exhibits fine-grained social behaviors in environments like SOTOPIA and Werewolf without human supervision, effectively collaborating even in competitive settings.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This paper introduces OMAR: One Model, All Roles, a reinforcement learning framework that enables AI to develop social intelligence through multi-turn, multi-agent conversational self-play. Unlike traditional paradigms that rely on static, single-turn optimizations, OMAR allows a single model to role-play all participants in a conversation simultaneously, learning to achieve long-term goals and complex social norms directly from dynamic social interaction. To ensure training stability across long dialogues, we implement a hierarchical advantage estimation that calculates turn-level and token-level advantages. Evaluations in the SOTOPIA social environment and Werewolf strategy games show that our trained models develop fine-grained, emergent social intelligence, such as empathy, persuasion, and compromise seeking, demonstrating the effectiveness of learning collaboration even under competitive scenarios. While we identify practical challenges like reward hacking, our results show that rich social intelligence can emerge without human supervision. We hope this work incentivizes further research on AI social intelligence in group conversations.

Problem

• social intelligence
• multi-agent conversation
• reinforcement learning
• multi-turn dialogue
• role-playing

Innovation

• multi-agent self-play
• conversational reinforcement learning
• social intelligence emergence
• hierarchical advantage estimation
• role-playing dialogue