Aligning Video Models with Human Social Judgments via Behavior-Guided Fine-Tuning

2025-10-01

AI Summary

This study investigates whether video-language models can capture the human perceptual structure of interactional similarity in social videos and proposes a behavior-guided alignment framework. To address the modality gap, we design a joint objective combining hybrid triplet loss with representational similarity analysis (RSA), optimized via fine-tuning on large-scale human pairwise similarity judgments. Leveraging a TimeSformer backbone, we incorporate LoRA for parameter-efficient adaptation. Experiments demonstrate that the model increases explained variance on held-out videos by 12.3%, improves triplet judgment accuracy by 9.8%, and significantly enhances representation fidelity for socio-affective attributes—including intimacy and valence. Our core contribution is the first systematic integration of human social perception behavioral data into the training objective of video foundation models, thereby bridging the gap between perceptual judgment and neural representation in social video understanding.

Technology Category

• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Humans intuitively perceive complex social signals in visual scenes, yet it remains unclear whether state-of-the-art AI models encode the same similarity structure. We study (Q1) whether modern video and language models capture human-perceived similarity in social videos, and (Q2) how to instill this structure into models using human behavioral data. To address this, we introduce a new benchmark of over 49,000 odd-one-out similarity judgments on 250 three-second video clips of social interactions, and discover a modality gap: despite the task being visual, caption-based language embeddings align better with human similarity than any pretrained video model. We close this gap by fine-tuning a TimeSformer video model on these human judgments with our novel hybrid triplet-RSA objective using low-rank adaptation (LoRA), aligning pairwise distances to human similarity. This fine-tuning protocol yields significantly improved alignment with human perceptions on held-out videos in terms of both explained variance and odd-one-out triplet accuracy. Variance partitioning shows that the fine-tuned video model increases shared variance with language embeddings and explains additional unique variance not captured by the language model. Finally, we test transfer via linear probes and find that human-similarity fine-tuning strengthens the encoding of social-affective attributes (intimacy, valence, dominance, communication) relative to the pretrained baseline. Overall, our findings highlight a gap in pretrained video models' social recognition and demonstrate that behavior-guided fine-tuning shapes video representations toward human social perception.

Problem

• Video models lack human-like social perception in videos
• AI models fail to capture human similarity judgments in social interactions
• Pretrained video models show modality gap in social recognition

Innovation

• Fine-tuning video models using human similarity judgments
• Hybrid triplet-RSA objective with low-rank adaptation
• Aligning video representations with human social perception