SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems

2026-04-17

AI Summary

This work addresses the limited planning and social reasoning capabilities of large language models in embodied multi-agent environments by introducing SocialGrid—a benchmark environment inspired by the game Among Us. The framework uniquely decouples task planning from social reasoning, enabling isolated evaluation of social intelligence through an optional planning oracle. It further supports systematic diagnosis via adversarial tournaments, fine-grained metrics, automated failure analysis, and an Elo-based leaderboard. Experimental results reveal that even the strongest open-source model (GPT-OSS-120B) achieves a task completion rate below 60% and frequently exhibits repetitive behaviors. Notably, its deception detection performance remains near random chance even with planning assistance, highlighting a critical bottleneck in current models’ social reasoning abilities.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings becomes critical. We introduce SocialGrid, an embodied multi-agent environment inspired by Among Us that evaluates LLM agents on planning, task execution, and social reasoning. Our evaluations reveal that even the strongest open model (GPT-OSS-120B) achieves below 60% accuracy in task completion and planning, with agents getting stuck in repetitive behaviors or failing to navigate basic obstacles. Since poor navigation confounds evaluation of social intelligence, SocialGrid offers an optional Planning Oracle to isolate social reasoning from planning deficits. While planning assistance improves task completion, social reasoning remains a bottleneck: agents fail to detect deception at near-random chance regardless of scale, relying on shallow heuristics rather than accumulating behavioral evidence. SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents. We also establish a competitive leaderboard using Elo ratings from adversarial league play.

Problem

• social reasoning
• embodied multi-agent systems
• planning
• deception detection
• Large Language Models

Innovation

• SocialGrid
• embodied multi-agent systems
• social reasoning
• planning oracle
• adversarial evaluation