SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

2025-05-29

AI Summary

Existing evaluation frameworks inadequately characterize large language models’ (LLMs) social reasoning capabilities, suffering from oversimplified scenarios, task-centric design, and insufficient modeling of dynamics and uncertainty. This paper introduces SocialMaze—the first systematic benchmark for social reasoning—spanning three authentic domains: social games, daily life, and digital communities. It targets three core challenges: deep reasoning, dynamic interaction, and information uncertainty. We propose a novel three-dimensional evaluation framework, comprising six diverse tasks integrating LLM-based automated assessment and human validation. Chain-of-thought (CoT) analysis identifies critical capability bottlenecks, guiding a targeted reasoning-sample fine-tuning method. Experiments reveal that strong CoT models excel at deep reasoning; temporal and interactive modeling significantly impact performance; and domain-specific fine-tuning substantially improves results on complex social tasks. The SocialMaze dataset is publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly applied to socially grounded tasks, such as online community moderation, media content analysis, and social reasoning games. Success in these contexts depends on a model's social reasoning ability - the capacity to interpret social contexts, infer others' mental states, and assess the truthfulness of presented information. However, there is currently no systematic evaluation framework that comprehensively assesses the social reasoning capabilities of LLMs. Existing efforts often oversimplify real-world scenarios and consist of tasks that are too basic to challenge advanced models. To address this gap, we introduce SocialMaze, a new benchmark specifically designed to evaluate social reasoning. SocialMaze systematically incorporates three core challenges: deep reasoning, dynamic interaction, and information uncertainty. It provides six diverse tasks across three key settings: social reasoning games, daily-life interactions, and digital community platforms. Both automated and human validation are used to ensure data quality. Our evaluation reveals several key insights: models vary substantially in their ability to handle dynamic interactions and integrate temporally evolving information; models with strong chain-of-thought reasoning perform better on tasks requiring deeper inference beyond surface-level cues; and model reasoning degrades significantly under uncertainty. Furthermore, we show that targeted fine-tuning on curated reasoning examples can greatly improve model performance in complex social scenarios. The dataset is publicly available at: https://huggingface.co/datasets/MBZUAI/SocialMaze

Problem

• Lack of systematic evaluation for LLMs' social reasoning abilities
• Existing benchmarks oversimplify real-world social scenarios
• Need for assessing deep reasoning, dynamic interaction, and uncertainty handling

Innovation

• Introduces SocialMaze benchmark for social reasoning
• Incorporates deep reasoning and dynamic interaction
• Uses automated and human validation for quality