Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

2026-08-10

AI Summary

Current large language models (LLMs) lack objective and verifiable benchmarks for evaluating their capabilities in cooperation, negotiation, and adaptation within multi-agent social scenarios. To address this gap, this work introduces Social Gym—a benchmark platform comprising 21 well-defined multi-agent social games—and establishes a cross-task leaderboard using Elo tournament scoring. Additionally, the authors propose SPaRTan, a training-free self-improvement method that leverages self-play, trajectory reflection, and zero-shot policy transfer. Experiments demonstrate that SPaRTan substantially enhances GPT-5-mini’s performance in disadvantaged roles, though its gains on Qwen3-32B are limited, highlighting both the method’s efficacy and its dependence on base model capabilities. This study presents the first reproducible, objective benchmark for LLM-based social reasoning alongside a parameter-free self-reflection mechanism.

Abstract

LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.

Problem

• social reasoning
• multi-agent systems
• LLM evaluation
• objective benchmarking
• social interaction

Innovation

• Social Gym
• SPaRTan
• multi-agent social reasoning
• Elo tournament
• training-free self-improvement