Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

2024-12-06

AI Summary

To address the performance limitations of open-source multimodal large language models (MLLMs) on complex vision-language understanding tasks, this work introduces the InternVL 2.5 series. Methodologically, it pioneers joint scaling of the visual encoder and language model, integrated with multi-stage alignment training, high-quality cross-modal data curation, test-time Chain-of-Thought reasoning, and dynamic ensemble re-ranking. The key contributions are threefold: (1) It achieves the first open-source MLLM result exceeding 70% on the MMMU benchmark (70.1%, +3.7 points), matching the performance of GPT-4o and Claude-3.5-Sonnet; (2) It systematically characterizes the joint scaling laws among visual encoder capacity, language model size, data scale, and test-time strategies; and (3) It provides the first empirical validation of strong test-time scaling effectiveness in open-source MLLMs. All models and an interactive Hugging Face demo are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see https://huggingface.co/spaces/OpenGVLab/InternVL

Problem

• Multimodal Modeling
• Visual Understanding
• Text Understanding

Innovation

• Multimodal Modeling
• Performance Enhancement
• Artificial Intelligence Advancement