BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

2025-05-14

AI Summary

This paper addresses the challenge of unifying image understanding and generation in multimodal models by proposing the first diffusion Transformer architecture that directly generates CLIP image features. Methodologically: (1) it abandons the VAE latent space and instead performs diffusion-based generation in the semantically rich CLIP feature space; (2) it introduces a two-stage pretraining paradigm—“understanding-first, then generation”—to jointly optimize comprehension and synthesis capabilities; (3) it leverages GPT-4o to automatically construct BLIP3o-60k, a high-quality instruction-tuning dataset. Contributions include state-of-the-art performance across major understanding benchmarks (VQAv2, OK-VQA) and generation metrics (FID, CLIP-Score); 37% higher training efficiency; significantly improved generation fidelity; and full-stack open-sourcing of code, model weights, training scripts, and data.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.

Problem

• Exploring optimal architecture for unified image understanding and generation
• Investigating training strategies to balance understanding and generation
• Creating high-quality datasets for multimodal instruction tuning

Innovation

• Uses diffusion transformer for CLIP image features
• Sequential pretraining for unified understanding and generation
• High-quality GPT-4o curated dataset BLIP3o-60k