Social Caption: Evaluating Social Understanding in Multimodal Models

2026-01-21

AI Summary

This work addresses the limited capability of multimodal large language models (MLLMs) in understanding human social interactions and the absence of systematic evaluation frameworks. To bridge this gap, the authors propose Social Caption, a novel benchmark grounded in interaction theory, which introduces the first three-dimensional evaluation framework encompassing social reasoning, holistic social analysis, and targeted social analysis. Through structured tasks and tailored metrics, the framework rigorously assesses models’ social comprehension abilities. Experimental results demonstrate that both model scale and architecture significantly influence social cognition performance, underscoring the framework’s effectiveness and innovation in advancing automated evaluation of social understanding in artificial intelligence systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding.

Problem

• Social Understanding
• Multimodal Models
• Social Inference
• Holistic Social Analysis
• Directed Social Analysis

Innovation

• Social Caption
• Multimodal Large Language Models
• Social Understanding Evaluation
• Interaction Theory
• Automated Evaluation