VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

2026-07-16

AI Summary

This work addresses the limitations of existing open-source video multimodal large language models, which often exhibit constrained generalization capabilities, suboptimal computational efficiency, and insufficient openness for diverse real-world applications. To overcome these challenges, the authors propose a fully open-source, efficient, and general-purpose video multimodal large language model. Key innovations include a novel Inflated 3D Vision Transformer (I3D-ViT) visual encoder, an adaptive frame-rate and resolution mechanism to enhance computational efficiency, and a large-scale synthetic dataset paired with a scalable training pipeline encompassing general, long-form, and streaming video scenarios. Despite having only 4 billion parameters, the model surpasses existing open-source counterparts—both comparable and larger in scale—across multiple benchmarks, achieving an exceptional balance between efficiency and strong generalization performance.

Abstract

Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.

Problem

• video understanding
• multimodal large language model
• generalization
• computational efficiency
• open-source

Innovation

• I3D-ViT
• Adaptive Frame Resolution
• Video Data Synthesis
• Fully Open MLLM
• Efficient Video Understanding