Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding

2025-05-19

AI Summary

This work addresses the weak temporal modeling capability of large vision-language models (LVLMs) in video understanding, systematically identifying for the first time the intermediate interface between the visual encoder and language model as the critical bottleneck for temporal reasoning. We propose a dual-path transfer paradigm: “interface upgrading + explicit temporal modeling.” Specifically, we design an Upscaled Interface—a scalable bridging module—incorporating cross-frame attention supervision and temporally enhanced sampling, coupled with a multi-stage progressive fine-tuning strategy. Our approach eliminates reliance on implicit temporal inference. Evaluated on standard benchmarks across action recognition, temporal localization, and video question answering, it achieves consistent improvements of 4.2–7.8% in accuracy, significantly enhancing LVLMs’ capacity for fine-grained temporal understanding in videos.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent years have witnessed outstanding advances of large vision-language models (LVLMs). In order to tackle video understanding, most of them depend upon their implicit temporal understanding capacity. As such, they have not deciphered important components that contribute to temporal understanding ability, which might limit the potential of these LVLMs for video understanding. In this work, we conduct a thorough empirical study to demystify crucial components that influence the temporal understanding of LVLMs. Our empirical study reveals that significant impacts are centered around the intermediate interface between the visual encoder and the large language model. Building on these insights, we propose a temporal-oriented recipe that encompasses temporal-oriented training schemes and an upscaled interface. Our final model developed using our recipe significantly enhances previous LVLMs on standard video understanding tasks.

Problem

• Identify key components for temporal understanding in LVLMs
• Enhance LVLMs' video understanding via temporal-oriented methods
• Improve interface between visual encoder and language model

Innovation

• Empirical study on LVLMs temporal understanding components
• Temporal-oriented training schemes for video understanding
• Upscaled interface between visual encoder and LLM