SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

2026-03-17

AI Summary

This work addresses the lack of effective evaluation for multimodal large language models in understanding and responding to dynamic social interactions. To this end, we propose SocialOmni—the first multimodal benchmark specifically designed for audio-visual social interaction—systematically assessing model performance along three dimensions: speaker-separated recognition, interruption timing control, and natural interruption generation. The framework introduces temporally and contextually constrained interactive generation samples, robustness tests under audio-visual inconsistency, and reveals a critical disconnect between perception accuracy and interactive generation capability. Experiments across twelve state-of-the-art models demonstrate that conventional perception-based metrics fail to reflect genuine social interaction competence, thereby validating the necessity and effectiveness of the proposed benchmark.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM benchmarks remain anchored to static, accuracy-centric tasks, leaving a critical gap in assessing social interactivity, the fundamental capacity to navigate dynamic cues in natural dialogues. To this end, we propose SocialOmni, a comprehensive benchmark that operationalizes the evaluation of this conversational interactivity across three core dimensions: (i) speaker separation and identification (who is speaking), (ii) interruption timing control (when to interject), and (iii) natural interruption generation (how to phrase the interruption). SocialOmni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances with strict temporal and contextual constraints, complemented by controlled audio-visual inconsistency scenarios to test model robustness. We benchmarked 12 leading OLMs, which uncovers significant variance in their social-interaction capabilities across models. Furthermore, our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions, indicating that understanding-centric metrics alone are insufficient to characterize conversational social competence. More encouragingly, these diagnostics from SocialOmni yield actionable signals for bridging the perception-interaction divide in future OLMs.

Problem

• social interactivity
• omni-modal large language models
• conversational interaction
• audio-visual understanding
• interruption generation

Innovation

• social interactivity
• omni-modal LLMs
• interruption generation
• speaker identification
• audio-visual benchmarking