VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models

2025-11-24

AI Summary

To address the challenge that Video Multimodal Large Language Models (VMLLMs) struggle to capture brief actions and rare transient events in long videos, this paper proposes a novel approach to enhance temporal fine-grained perception. The method introduces three key innovations: (1) synthesizing video samples with deliberately omitted keyframes to emphasize transient dynamics; (2) a two-stage training framework integrating supervised fine-tuning with reinforcement learning using frame-replacement negative samples; and (3) a relative reward mechanism coupled with an auxiliary contrastive loss to explicitly align visual intermediate representations with semantic keywords. Experimental results demonstrate significant improvements over state-of-the-art methods on fine-grained action understanding and rare event description tasks, while maintaining competitive performance on standard video understanding benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We propose VideoPerceiver, a novel video multimodal large language model (VMLLM) that enhances fine-grained perception in video understanding, addressing VMLLMs' limited ability to reason about brief actions in short clips or rare transient events in long videos. VideoPerceiver adopts a two-stage training framework. During supervised fine-tuning (SFT), we construct "key-information-missing" videos by extracting event-action keywords from captions, identifying corresponding key frames, and replacing them with adjacent frames. We jointly encode original and modified video tokens with text tokens, aligning intermediate visual representations with keywords via an auxiliary contrastive loss to enhance sensitivity to fine-grained motion cues. In reinforcement learning (RL), both video variants are fed into the model to generate descriptions, and a novel relative reward ensures responses from complete videos outperform those from degraded inputs, explicitly training the model to recover temporally precise action details. We also curate a dataset of 80,000 videos with fine-grained actions and transient events. Experiments show VideoPerceiver substantially outperforms state-of-the-art VMLLMs on fine-grained action understanding and rare event captioning benchmarks, while maintaining strong performance on standard tasks. By prioritizing task-relevant visual features, our work redefines video-language model training for fine-grained perception.

Problem

• Addressing VMLLMs' limited reasoning about brief actions in short clips
• Improving detection of rare transient events in long video sequences
• Enhancing sensitivity to fine-grained motion cues in video understanding

Innovation

• Two-stage training with key-frame replacement
• Auxiliary contrastive loss for motion sensitivity
• Relative reward for precise action recovery