Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

2026-01-29

AI Summary

This work proposes UniMRG, a unified multimodal framework that addresses the limited synergy between visual understanding and generation in existing models. By integrating auxiliary generative tasks—such as pixel reconstruction, depth estimation, and semantic segmentation—within a single architecture, UniMRG enables bidirectional enhancement between comprehension and synthesis. The method employs an architecture-agnostic post-training strategy, uniquely leveraging multitask generation to retroactively improve visual understanding capabilities. Experimental results demonstrate that UniMRG significantly advances performance in fine-grained perception, spatial relationship modeling, and hallucination suppression, while simultaneously enhancing generation quality. These findings validate the efficacy of the proposed understanding-generation co-evolution mechanism within a unified model.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.

Problem

• Unified Multimodal Models
• visual understanding
• generation
• multi-representation
• post-training

Innovation

• Unified Multimodal Models
• Multi-Representation Generation
• Visual Understanding
• Post-Training
• Auxiliary Generation Tasks