WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysis

2024-12-04

arXiv.org

AI Summary

Existing LLM-driven multi-agent system (MAS) evaluations suffer from poor reproducibility, coarse-grained analysis, and difficulties in cross-model comparison. To address these challenges, this work introduces an open-source evaluation platform grounded in the “Who is Spy?” game, pioneering a gamified, dynamic evaluation paradigm. The platform enables unified integration of both open- and closed-weight LLMs and supports fine-grained, strategy-level quantification—specifically across attack, defense, and reasoning behaviors. It incorporates Hugging Face model interfacing, a real-time web leaderboard, a configurable rule engine, interactive log analytics, and win-rate attribution modeling. Empirical validation across 20+ state-of-the-art LLMs demonstrates the platform’s efficacy in uncovering significant behavioral disparities in collaboration, deception, and logical reasoning. Consequently, it substantially enhances interpretability, reproducibility, and横向 comparability of MAS evaluations.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in autonomous multi-agent systems (MAS) based on large language models (LLMs) have enhanced the application scenarios and improved the capability of LLMs to handle complex tasks. Despite demonstrating effectiveness, existing studies still evidently struggle to evaluate, analysis, and reproducibility of LLM-based MAS. In this paper, to facilitate the research on LLM-based MAS, we introduce an open, scalable, and real-time updated platform for accessing and analyzing the LLM-based MAS based on the games Who is Spy?"(WiS). Our platform is featured with three main worths: (1) a unified model evaluate interface that supports models available on Hugging Face; (2) real-time updated leaderboard for model evaluation; (3) a comprehensive evaluation covering game-winning rates, attacking, defense strategies, and reasoning of LLMs. To rigorously test WiS, we conduct extensive experiments coverage of various open- and closed-source LLMs, we find that different agents exhibit distinct and intriguing behaviors in the game. The experimental results demonstrate the effectiveness and efficiency of our platform in evaluating LLM-based MAS. Our platform and its documentation are publicly available at url{https://whoisspy.ai/}

Problem

• Evaluating LLM-based multi-agent systems effectively
• Analyzing game strategies and reasoning in LLMs
• Providing a scalable platform for MAS reproducibility

Innovation

• Game-based platform for LLM multi-agent evaluation
• Real-time leaderboard and unified model interface
• Comprehensive metrics for agent behavior analysis