Multi-Party Conversational Agents: A Survey

2025-05-24

AI Summary

This paper addresses the design challenges of multi-party conversational agents (MPCAs), which require simultaneous modeling of participants’ mental states, semantic understanding, and behavioral prediction. We adopt Theory of Mind (ToM) as the foundational paradigm and systematically survey three core challenges—mental state modeling, semantic comprehension, and action decision-making—tracing the technical evolution from conventional models to large language models (LLMs) and multimodal integration. We introduce, for the first time, a three-dimensional evaluation framework encompassing sociality, linguistic competence, and interactivity, identifying critical bottlenecks in current approaches. Our analysis underscores multimodal understanding as a pivotal unresolved direction. The work establishes a theoretical foundation and a systematic roadmap for developing socially intelligent, group-level dialogue systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Multiagent Systems: Modeling other Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Multi-party Conversational Agents (MPCAs) are systems designed to engage in dialogue with more than two participants simultaneously. Unlike traditional two-party agents, designing MPCAs faces additional challenges due to the need to interpret both utterance semantics and social dynamics. This survey explores recent progress in MPCAs by addressing three key questions: 1) Can agents model each participants' mental states? (State of Mind Modeling); 2) Can they properly understand the dialogue content? (Semantic Understanding); and 3) Can they reason about and predict future conversation flow? (Agent Action Modeling). We review methods ranging from classical machine learning to Large Language Models (LLMs) and multi-modal systems. Our analysis underscores Theory of Mind (ToM) as essential for building intelligent MPCAs and highlights multi-modal understanding as a promising yet underexplored direction. Finally, this survey offers guidance to future researchers on developing more capable MPCAs.

Problem

• Modeling participants' mental states in multi-party dialogues
• Understanding semantic content in group conversations
• Predicting and reasoning about future conversation dynamics

Innovation

• Modeling mental states using Theory of Mind
• Leveraging Large Language Models for semantics
• Multi-modal systems for dynamic understanding