Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment

2025-02-06

AI Summary

Current state-of-the-art multimodal large language models (MLLMs) exhibit substantially inferior performance in image, video, and audio understanding compared to specialized unimodal models. To address this gap, we introduce Ola—the first open-source, unified multimodal language model supporting joint comprehension of text, images, audio, and video. Methodologically, we propose a progressive multimodal alignment paradigm: starting from a vision–language foundation, we incrementally incorporate speech and video modalities; design a sentence-level streaming speech decoding mechanism for low-latency real-time interaction; and integrate cross-modal adapters with a lightweight, high-quality alignment data curation strategy. Extensive experiments demonstrate that Ola consistently outperforms leading open-source MLLMs across all three modalities—image, video, and audio understanding—while matching the performance of similarly sized unimodal specialist models. The model weights, training code, and alignment datasets are fully open-sourced.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in large language models, particularly following GPT-4o, have sparked increasing interest in developing omni-modal models capable of understanding more modalities. While some open-source alternatives have emerged, there is still a notable lag behind specialized single-modality models in performance. In this paper, we present Ola, an Omni-modal language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts. The core design of Ola lies in its progressive modality alignment strategy that extends the supporting modality of the language model progressively. Our training pipeline begins with the most distinct modalities: image and text, then gradually expands the skill sets of the model using speech data that connects language and audio knowledge, and video data that connects all modalities. The progressive learning pipeline also enables us to maintain a relatively small size of the cross-modal alignment data, making developing omni-modal from existing vision-language models easy and less costly. Moreover, to unlock an advanced interactive experience like GPT-4o, we further design a sentence-wise decoding solution for streaming speech generation. Extensive experiments demonstrate that Ola surpasses existing open omni-modal LLMs across all modalities while achieving highly competitive performance compared to state-of-the-art specialized models of similar sizes. We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field. Model weights, code, and data are open-sourced at https://github.com/Ola-Omni/Ola.

Problem

• developing competitive omni-modal language models
• progressive modality alignment strategy
• enhancing multi-modal understanding performance

Innovation

• Progressive modality alignment strategy
• Sentence-wise decoding for speech
• Small cross-modal alignment data