Questioning the Unknown: Optimising Multi-Agent Collaboration in Narrative-Driven Games

2024-04-26

AI Summary

To address three core challenges in Murder Mystery Games (MMGs) for LLM-based agents—undefined state space, absence of intermediate rewards, and difficulty in strategic interaction under continuous linguistic input—this paper proposes Questum. Methodologically, Questum introduces: (1) sensor-inspired dynamic state representation to explicitly model implicit game states; (2) an information-gain-driven questioning mechanism coupled with suspect list pruning to enhance reasoning efficiency; and (3) WellPlay, the first MMG-specific reasoning evaluation benchmark comprising 1,482 inferential questions. Experimental results demonstrate that Questum significantly outperforms baseline methods in reasoning accuracy, response efficiency, and naturalness and trustworthiness of human-agent collaboration. By unifying explicit state modeling, principled information acquisition, and rigorous evaluation, Questum establishes a novel paradigm for language-driven multi-agent strategic reasoning.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Multiagent Systems: Mechanism Design

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

We present Questum, a novel framework for Large Language Model (LLM)-based agents in Murder Mystery Games (MMGs). MMGs pose unique challenges, including undefined state spaces, absent intermediate rewards, and the need for strategic interaction in a continuous language domain. Questum addresses these complexities through a sensor-based representation of agent states, a question-targeting mechanism guided by information gain, and a pruning strategy to refine suspect lists and enhance decision-making efficiency. To enable systematic evaluation, we propose WellPlay, a dataset comprising 1,482 inferential questions across 12 games, categorised into objectives, reasoning, and relationships. Experiments demonstrate Questum's capacity to achieve superior performance in reasoning accuracy and efficiency compared to existing approaches, while also significantly improving the quality of agent-human interactions in MMGs. This study advances the development of reasoning agents for complex social and interactive scenarios.

Problem

• Enhancing multi-agent communication in murder mystery games
• Addressing undefined state spaces and strategic interaction challenges
• Improving reasoning accuracy and agent-human interaction quality

Innovation

• Sensor-based representation of agent states
• Question-targeting mechanism using information gain
• Pruning strategy for suspect list refinement