HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

2024-12-20

arXiv.org

AI Summary

Existing monolithic vision-language models (VLMs) suffer from a lack of unified vision-language embedding modules, hindering simultaneous achievement of strong multimodal capabilities and native large language model (LLM) linguistic performance. To address this, we propose HoVLE—a high-performance monolithic VLM that preserves the original LLM architecture while enabling native multimodal understanding. Its core innovation is a novel all-modal joint embedding module that maps visual and textual inputs into a shared semantic space. HoVLE employs a three-stage training paradigm: (1) vision-text feature distillation, (2) cross-modal contrastive alignment, and (3) instruction fine-tuning. On benchmarks including MMBench and OCRBench, HoVLE matches or exceeds the performance of prominent modular VLMs (e.g., Qwen-VL, LLaVA) and significantly outperforms prior monolithic models (e.g., KOSMOS-2, mPLUG-Owl2). The model is publicly released on Hugging Face.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid advance of Large Language Models (LLMs) has catalyzed the development of Vision-Language Models (VLMs). Monolithic VLMs, which avoid modality-specific encoders, offer a promising alternative to the compositional ones but face the challenge of inferior performance. Most existing monolithic VLMs require tuning pre-trained LLMs to acquire vision abilities, which may degrade their language capabilities. To address this dilemma, this paper presents a novel high-performance monolithic VLM named HoVLE. We note that LLMs have been shown capable of interpreting images, when image embeddings are aligned with text embeddings. The challenge for current monolithic VLMs actually lies in the lack of a holistic embedding module for both vision and language inputs. Therefore, HoVLE introduces a holistic embedding module that converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts. Furthermore, a multi-stage training strategy is carefully designed to empower the holistic embedding module. It is first trained to distill visual features from a pre-trained vision encoder and text embeddings from the LLM, enabling large-scale training with unpaired random images and text tokens. The whole model further undergoes next-token prediction on multi-modal data to align the embeddings. Finally, an instruction-tuning stage is incorporated. Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks, outperforming previous monolithic models by a large margin. Model available at https://huggingface.co/OpenGVLab/HoVLE.

Problem

• Improving monolithic Vision-Language Models
• Holistic embedding for vision and language
• Multi-stage training strategy for alignment

Innovation

• Holistic embedding module
• Multi-stage training strategy
• Shared space conversion