Human Semantic Representations of Social Interactions from Moving Shapes

2025-09-24

AI Summary

How do humans recognize social interactions from simple moving shapes, and what cognitive mechanisms—beyond low-level visual features—underlie this ability? Method: Using representational geometry, we systematically compared human similarity judgments against models based on low-level visual features, hand-labeled categories, and semantic embeddings derived from descriptive text—with a focus on verb-based embeddings capturing interaction semantics (e.g., “chasing”, “cooperating”). Contribution/Results: A verb-centric semantic embedding model best accounted for human judgments, significantly outperforming visual and alternative semantic baselines. Crucially, it provided not merely complementary but uniquely explanatory variance, revealing that abstract action-verb semantics constitute a core cognitive dimension driving social perception. This study provides the first empirical demonstration that verb-level semantic representations dominate social interpretation of minimal dynamic stimuli, offering critical evidence for cross-modal representational theories of social perception.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Natural Language Processing: Lexical Semantics and Morphology

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Social recommender systems and personalization for the social Web
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Humans are social creatures who readily recognize various social interactions from simple display of moving shapes. While previous research has often focused on visual features, we examine what semantic representations that humans employ to complement visual features. In Study 1, we directly asked human participants to label the animations based on their impression of moving shapes. We found that human responses were distributed. In Study 2, we measured the representational geometry of 27 social interactions through human similarity judgments and compared it with model predictions based on visual features, labels, and semantic embeddings from animation descriptions. We found that semantic models provided complementary information to visual features in explaining human judgments. Among the semantic models, verb-based embeddings extracted from descriptions account for human similarity judgments the best. These results suggest that social perception in simple displays reflects the semantic structure of social interactions, bridging visual and abstract representations.

Problem

• Examining semantic representations humans use to understand social interactions
• Comparing human similarity judgments with visual and semantic model predictions
• Identifying how verb-based embeddings best explain social perception judgments

Innovation

• Used verb-based semantic embeddings from descriptions
• Combined semantic models with visual features
• Measured representational geometry through similarity judgments