Transferability Between Understanding and Generation in Unified Multimodal Models

2026-07-05

AI Summary

This work addresses the unclear transferability mechanism between image understanding and generation tasks in existing unified multimodal models. It presents the first systematic investigation into cross-task transfer patterns between these two objectives and reveals that a shared Transformer backbone combined with a unified visual encoder enables stable knowledge transfer. Building on this insight, the study proposes a novel paradigm that enhances generative capabilities indirectly through training on understanding tasks, effectively mitigating distribution shift issues. The approach is validated on three critical competencies—counting, spatial reasoning, and text recognition/generation—demonstrating significant improvements in generation performance without compromising visual fidelity.

Abstract

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Problem

• transferability
• unified multimodal models
• image understanding
• image generation
• cross-task interaction

Innovation

• transferability
• unified multimodal models
• cross-task learning
• shared transformer backbone
• distribution shift