VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding

2025-01-22

AI Summary

To address the performance fragmentation between image and video understanding tasks and the underutilization of multimodal data, this paper introduces VideoLLaMA3, a vision-centric multimodal foundation model. Methodologically, it proposes a four-stage collaborative training paradigm: (1) vision alignment, (2) vision–language pretraining, (3) multi-task fine-tuning, and (4) video-specific fine-tuning. Key technical innovations include a vision-prioritized architecture, variable-resolution image encoding, similarity-driven video token compression, and a unified fusion mechanism for image, video, and text data. The contributions are threefold: (1) establishing a “vision-centric” modeling paradigm for multimodal learning; (2) achieving a balance between fine-grained visual perception and computational efficiency; and (3) empirically validating the critical role of high-quality image data in enhancing video understanding capability. VideoLLaMA3 achieves state-of-the-art performance across multiple image and video understanding benchmarks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of"vision-centric"is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) vision-centric alignment stage, which warms up the vision encoder and projector; 2) vision-language pretraining stage, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) multi-task fine-tuning stage, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) video-centric fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.

Problem

• Multi-modal Foundation Model
• Video Understanding
• Image Understanding

Innovation

• vision-centric paradigm
• multi-stage training
• adaptive tokenization