From Image to Video, what do we need in multimodal LLMs?

2024-04-18

arXiv.org

AI Summary

Training video large language models (Video LLMs) incurs prohibitively high computational costs and relies heavily on massive video datasets. Method: This paper proposes RED-VILLM—a resource-efficient development paradigm that leverages off-the-shelf image large language models (Image LLMs) and introduces a lightweight temporal adaptation module to enable efficient knowledge transfer, eliminating redundant architectural design and costly large-scale video pretraining. It achieves, for the first time, efficient cross-modal transfer from Image LLMs to Video LLMs via a plug-and-play temporal modeling structure, integrated with instruction tuning and multi-stage alignment training. Contribution/Results: RED-VILLM attains superior performance over conventional Video LLMs under extremely limited instruction data and computational resources, significantly reducing both computational overhead and dependency on video data. Additionally, this work releases the first open-source Video LLM tailored for the Chinese research community.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Covering from Image LLMs to the more complex Video LLMs, the Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in comprehending cross-modal information as numerous studies have illustrated. Previous methods delve into designing comprehensive Video LLMs through integrating video foundation models with primitive LLMs. Despite its effectiveness, such paradigm renders Video LLM's structure verbose and typically requires substantial video data for pre-training. Crucially, it neglects leveraging the foundational contributions of ready-made Image LLMs. In this paper, we introduce RED-VILLM, a Resource-Efficient Development pipeline which builds robust Video LLMs through leveraging the prior knowledge of Image LLMs. Specifically, since a video is naturally a combination of images along the temporal dimension, we devise a temporal adaptation plug-and-play structure, endowing the backbone Image LLM with the capability to grasp temporal information. Moreover, through applying this pipeline, we achieve the first Video LLM within the Chinese-speaking community. Extensive experiments demonstrate that Video LLMs developed through our approach surpass conventional Video LLMs, requiring minimal instructional data and training resources. Our approach highlights the potential for a more cost-effective and scalable advancement in multimodal models.

Problem

• Efficiently develop Video LLMs using Image LLMs
• Leverage Image LLMs' knowledge for temporal video understanding
• Minimize training data and resources for Video LLMs

Innovation

• Leverages Image LLMs for video understanding
• Introduces temporal adaptation plug-and-play structure
• Requires minimal data and training resources