Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification

2026-05-27

AI Summary

This work addresses the limitations of existing video key人物 recognition methods, which often neglect temporal context and erroneously downweight early yet crucial人物 due to temporal importance shifts. To tackle this, the study introduces the novel task of Video Important Person (VIP) recognition, accompanied by the Temporal-VIP dataset comprising 9,249 video clips annotated with human-justified rationales. The authors propose VIP-Net, a multimodal framework that integrates social cues, temporal dynamics, and cross-modal alignment through a Social Cue Encoder, a Temporal Importance Rectifier, and a cross-modal fusion mechanism to jointly infer the most influential person and generate explanatory text. Experiments demonstrate that VIP-Net achieves an accuracy of 67.3%, substantially outperforming state-of-the-art methods (37.5%–53.9%), with generated rationales attaining an average similarity of 0.63 against human annotations.

Abstract

Identifying key individuals in video scenes is essential for applications such as automated video editing and intelligent surveillance. Current methods primarily focus on static images and immediate visual cues, overlooking the rich spatio-temporal information in videos. This leads to the phenomenon of Temporal Importance Shift (TIS), wherein individuals deemed significant in early frames may be demoted as the entire temporal context is considered. To address this, we introduce the Video Important Person (VIP) identification task, aimed at automatically identifying the most influential individuals in videos while providing textual rationales. We present Temporal-VIP, a large-scale rationale-annotated dataset consisting of 9,249 video segments across 11 categories with aligned importance rationales. To mitigate TIS, we develop the VIP-Net framework, which includes a Social Cue Encoder (SCE) for extracting multi-modal spatio-temporal cues, a Temporal Importance Rectifier (TIR) for hierarchical cue fusion and cross-modal alignment, and VIP Inference for ranking individuals. Experimental results show that VIP-Net achieves 67.3% accuracy, significantly outperforming state-of-the-art models (37.5%-53.9%) and yielding a mean rationale similarity of 0.63 to ground truth through feature-guided LLM refinement. The dataset and code are available at https://huggingface.co/datasets/yml2002/Temporal-VIP.

Problem

• Video Important Person Identification
• Temporal Importance Shift
• Spatio-Temporal Cues
• Multi-Modality
• Key Individual Recognition

Innovation

• Temporal Importance Shift
• Multi-Modality Spatio-Temporal Cues
• VIP-Net
• Rationale-Guided Inference
• Video Important Person Identification