ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement

2025-04-02

AI Summary

Existing unified multimodal models exhibit performance imbalances across visual understanding, image generation, and image editing—suffering from insufficient semantic depth, low texture fidelity, and weak capability in text-image interleaved tasks. This paper proposes ILLUME+ (3B), a unified multimodal large language model framework featuring DualViTok, the first dual visual tokenizer that jointly models fine-grained texture and text-aligned semantics. We further introduce a diffusion-based decoder for high-fidelity detokenization and dynamic super-resolution, and adopt a progressive dynamic-resolution training paradigm with continuous inputs and discrete outputs. Evaluated on comprehensive benchmarks spanning understanding, generation, and editing, ILLUME+ achieves state-of-the-art or on-par performance with task-specific models. It is the first unified framework to enable context-aware, high-fidelity image editing and multi-scale controllable generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We present ILLUME+ that leverages dual visual tokenization and a diffusion decoder to improve both deep semantic understanding and high-fidelity image generation. Existing unified models have struggled to simultaneously handle the three fundamental capabilities in a unified model: understanding, generation, and editing. Models like Chameleon and EMU3 utilize VQGAN for image discretization, due to the lack of deep semantic interaction, they lag behind specialist models like LLaVA in visual understanding tasks. To mitigate this, LaViT and ILLUME employ semantic encoders for tokenization, but they struggle with image editing due to poor texture preservation. Meanwhile, Janus series decouples the input and output image representation, limiting their abilities to seamlessly handle interleaved image-text understanding and generation. In contrast, ILLUME+ introduces a unified dual visual tokenizer, DualViTok, which preserves both fine-grained textures and text-aligned semantics while enabling a coarse-to-fine image representation strategy for multimodal understanding and generation. Additionally, we employ a diffusion model as the image detokenizer for enhanced generation quality and efficient super-resolution. ILLUME+ follows a continuous-input, discrete-output scheme within the unified MLLM and adopts a progressive training procedure that supports dynamic resolution across the vision tokenizer, MLLM, and diffusion decoder. This design allows for flexible and efficient context-aware image editing and generation across diverse tasks. ILLUME+ (3B) exhibits competitive performance against existing unified MLLMs and specialized models across multimodal understanding, generation, and editing benchmarks. With its strong performance, ILLUME+ provides a scalable and versatile foundation for future multimodal applications. Project Page: https://illume-unified-mllm.github.io/.

Problem

• Improve deep semantic understanding and high-fidelity image generation
• Address limitations in unified models for understanding, generation, and editing
• Enhance texture preservation and semantic alignment in visual tokenization

Innovation

• Dual visual tokenization for unified understanding and generation
• Diffusion decoder enhances image generation quality
• Progressive training supports dynamic resolution tasks