InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling

2025-01-21

AI Summary

To address the challenge of simultaneously achieving long-term temporal modeling and fine-grained visual understanding in video multimodal large language models (MLLMs), this paper proposes the Long-and-Rich Context (LRC) paradigm. Methodologically, LRC integrates dense vision-task annotations (e.g., object tracking, segmentation) with adaptive hierarchical spatiotemporal token compression, and employs direct preference optimization (DPO) to align model outputs with human perceptual preferences. This design preserves high temporal resolution while significantly reducing representational redundancy, thereby enhancing long-horizon memory retention and attention focus. Experiments demonstrate state-of-the-art performance across both short- and long-video understanding benchmarks; support video inputs up to six times longer than those handled by existing methods; and—critically—enable, for the first time, end-to-end joint understanding of professional vision tasks within an MLLM framework.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper aims to improve the performance of video multimodal large language models (MLLM) via long and rich context (LRC) modeling. As a result, we develop a new version of InternVideo2.5 with a focus on enhancing the original MLLMs' ability to perceive fine-grained details and capture long-form temporal structure in videos. Specifically, our approach incorporates dense vision task annotations into MLLMs using direct preference optimization and develops compact spatiotemporal representations through adaptive hierarchical token compression. Experimental results demonstrate this unique design of LRC greatly improves the results of video MLLM in mainstream video understanding benchmarks (short&long), enabling the MLLM to memorize significantly longer video inputs (at least 6x longer than the original), and master specialized vision capabilities like object tracking and segmentation. Our work highlights the importance of multimodal context richness (length and fineness) in empowering MLLM's innate abilites (focus and memory), providing new insights for future research on video MLLM. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5

Problem

• Video Multimodal Large Language Model
• Long Video Information Processing
• Middle School Student Comprehension Enhancement

Innovation

• InternVideo2.5
• Video Multimodal Large Language Model (MLLM)
• Enhanced Temporal and Spatial Processing