Semantic Generative Tuning for Unified Multimodal Models

2026-05-18

AI Summary

This work addresses the misalignment between visual understanding and generation in unified multimodal models, which arises from disjoint training objectives and hinders their synergistic improvement. To bridge this gap, the authors propose Semantic Generation Tuning (SGT), a novel generative post-training paradigm that leverages image segmentation as an optimal high-level semantic proxy task to jointly align and optimize both capabilities within a single architecture. By using structured semantic signals to guide vision–language attention allocation, SGT enhances feature linear separability and significantly improves both visual understanding accuracy and generation layout fidelity across major multimodal benchmarks, achieving mutual enhancement of comprehension and synthesis abilities.

Abstract

Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture. However, prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives. Such a decoupled strategy yields misaligned representation spaces, isolating visual understanding from generation and hindering their mutual reinforcement. This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs. Our empirical investigation reveals that high-level semantic tasks, particularly image segmentation, serve as optimal proxies. Unlike low-level tasks that distract models with texture details, segmentation provides structural semantics that significantly enhance both vision-centric perception and generative layout fidelity. Building upon these insights, we introduce Semantic Generative Tuning (SGT), a novel paradigm that leverages segmentation as a generative proxy to align and synergize multimodal capabilities. Mechanistic analyses further demonstrate that SGT fundamentally improves feature linear separability and optimizes visual-textual attention allocation pattern. Extensive evaluations show that SGT consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. Our code is available on the https://song2yu.github.io/SGT/.

Problem

• unified multimodal models
• visual understanding
• visual generation
• representation alignment
• generative post-training

Innovation

• Semantic Generative Tuning
• Unified Multimodal Models
• Generative Post-Training
• Image Segmentation as Proxy
• Visual-Textual Alignment