NVILA: Efficient Frontier Visual Language Models

2024-12-05

arXiv.org

AI Summary

Current vision-language models (VLMs) achieve notable accuracy gains, yet high-resolution image and long-video understanding remain hindered by training and deployment inefficiencies. This paper introduces NVILA, an open-source VLM series that pioneers a “scale-then-compress” architectural paradigm to holistically optimize training, fine-tuning, and inference. Key innovations include VILA-enhanced visual token scaling and compression, efficient prefilling and decoding scheduling strategies, and a lightweight spatiotemporal modeling design. Experiments demonstrate that NVILA matches or surpasses leading open- and closed-source models in multiple vision-language understanding benchmarks. Moreover, it reduces training cost by 4.5×, fine-tuning GPU memory consumption by 3.4×, prefilling latency by 1.6–2.2×, and decoding latency by 1.2–2.8×—delivering substantial end-to-end efficiency improvements across the model lifecycle.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual language models (VLMs) have made significant advances in accuracy in recent years. However, their efficiency has received much less attention. This paper introduces NVILA, a family of open VLMs designed to optimize both efficiency and accuracy. Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens. This"scale-then-compress"approach enables NVILA to efficiently process high-resolution images and long videos. We also conduct a systematic investigation to enhance the efficiency of NVILA throughout its entire lifecycle, from training and fine-tuning to deployment. NVILA matches or surpasses the accuracy of many leading open and proprietary VLMs across a wide range of image and video benchmarks. At the same time, it reduces training costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by 1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and models available to facilitate reproducibility.

Problem

• Optimizes efficiency and accuracy of visual language models.
• Enhances processing of high-resolution images and long videos.
• Reduces training costs and improves deployment efficiency.

Innovation

• Scale-then-compress visual token processing
• Systematic lifecycle efficiency enhancement
• Reduced training and deployment costs