OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment

2025-09-23

AI Summary

Current multimodal large models typically handle understanding, generation, and retrieval tasks separately, leading to high computational overhead and weak cross-modal generalization. To address this, we propose the first lightweight multimodal framework enabling unified modeling of all three tasks: it leverages a pre-trained language model (LLM) as the central language backbone, introduces a learnable bidirectional latent space alignment module, and employs a two-stage decoupled training strategy combined with semantic-guided diffusion training to mitigate task interference. Crucially, our framework achieves end-to-end visual–language representation alignment within a shared latent space—without requiring from-scratch training. Extensive experiments demonstrate state-of-the-art or competitive performance across multiple benchmarks, validating the efficacy of latent space alignment for unified multimodal modeling. The code and models are publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in multimodal large language models (LLMs) have led to significant progress in understanding, generation, and retrieval tasks. However, current solutions often treat these tasks in isolation or require training LLMs from scratch, resulting in high computational costs and limited generalization across modalities. In this work, we present OmniBridge, a unified and modular multimodal framework that supports vision-language understanding, generation, and retrieval within a unified architecture. OmniBridge adopts a language-centric design that reuses pretrained LLMs and introduces a lightweight bidirectional latent alignment module. To address the challenge of task interference, we propose a two-stage decoupled training strategy: supervised fine-tuning and latent space alignment for aligning LLM behavior with multimodal reasoning, and semantic-guided diffusion training to align cross-modal latent spaces via learnable query embeddings. Extensive experiments across a wide range of benchmarks demonstrate that OmniBridge achieves competitive or state-of-the-art performance in all three tasks. Moreover, our results highlight the effectiveness of latent space alignment for unifying multimodal modeling under a shared representation space. Code and models are released at https://github.com/xiao-xt/OmniBridge.

Problem

• Unifying multimodal understanding, generation, and retrieval tasks
• Addressing task interference and limited cross-modal generalization
• Reducing computational costs by reusing pretrained language models

Innovation

• Lightweight bidirectional latent alignment module
• Two-stage decoupled training strategy
• Semantic-guided diffusion with query embeddings