Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

2024-08-22

International Conference on Learning Representations

AI Summary

To address the limitation of requiring multiple specialized models for multimodal understanding and generation tasks, this paper proposes Show-o—the first unified Transformer architecture capable of both understanding and generation. Its core innovation is a novel paradigm that jointly integrates autoregressive modeling with discretized diffusion mechanisms, enabling flexible input/output configurations across arbitrary modality combinations (e.g., image-text pairs) and token-level cross-modal joint modeling. Unlike prior approaches, Show-o eliminates task-specific architectural designs. Evaluated on diverse benchmarks—including visual question answering, text-to-image synthesis, text-guided inpainting, and extrapolation—Show-o achieves performance on par with or superior to dedicated large models, despite using equal or fewer parameters. The framework demonstrates strong generalization across modalities and tasks without architectural modifications. Code and pretrained models are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at https://github.com/showlab/Show-o.

Problem

• Unifies multimodal understanding and generation tasks
• Handles inputs and outputs of various mixed modalities
• Supports vision-language tasks like question-answering and generation

Innovation

• Unifies multimodal understanding and generation
• Combines autoregressive and discrete diffusion modeling
• Handles inputs and outputs of mixed modalities