Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation

2026-06-13

AI Summary

This work addresses the limitation of existing social intelligence benchmarks, which predominantly focus on textual modalities while neglecting the critical role of visual cues in social interaction. To bridge this gap, the authors introduce SocialBench, the first multimodal social simulation benchmark comprising 240 scenarios, 585 characters, and 2,340 tasks. SocialBench systematically evaluates agents’ visual social competencies across four hierarchical role-based task categories—facial expression, personality manifestation, interaction regulation, and outcome achievement—leveraging image-text aligned evidence and structured character profiles. Evaluations of multimodal large language models (MLLMs) under both verbalized-vision and direct-vision paradigms reveal that while current models nearly saturate performance in character-specific expression and conflict handling, they exhibit significant deficiencies in interaction regulation and outcome achievement when these tasks rely on visual cues, thereby uncovering substantial challenges in high-level visual social reasoning.

Abstract

Social interaction depends on both language and visible social signals, such as facial expressions, posture, gaze, and emotional shifts. Yet existing social-agent benchmarks are largely text-based and rarely test whether multimodal agents can use visual cues to guide interaction. We introduce \textsc{\benchmarkname{}}, a benchmark evaluating visual social intelligence in multimodal social simulation. It contains 240 scenarios, 585 role instances, and 2,340 role-task instances, combining aligned textual-visual evidence, structured role profiles, and four role-level tasks: expression task, characteristic task, interaction regulation task, and interaction outcome task. Evaluating seven recent MLLMs under verbalized-vision and direct-vision reveals a clear gap between local role enactment and interaction management: role-specific expression and conflict handling are near saturation, whereas interaction regulation and visually grounded outcome achievement remain substantially more difficult. The code is released at https://github.com/JunsWan/AgentViSS, and the dataset is available at https://huggingface.co/datasets/JunsWan/AgentViSS.

Problem

• visual social intelligence
• multimodal simulation
• social interaction
• visual cues
• social-agent benchmark

Innovation

• visual social intelligence
• multimodal simulation
• social-agent benchmark
• role-level tasks
• MLLMs