Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners

2026-05-14

AI Summary

This work addresses the attention entanglement and visual refinement bottlenecks in unified multimodal models for any-to-image (X2I) generation, which stem from the disconnection between comprehension and generation. To this end, we propose an adaptive interleaved visual reasoning framework that constructs a hierarchical data pipeline, enabling the model to dynamically switch among three modes—direct generation, self-reflection, and multi-step planning—based on instruction complexity. The framework employs a two-stage training strategy combining supervised fine-tuning (SFT) and reinforcement learning (RL), augmented with step-wise reasoning rewards and intra-batch complexity penalties to ensure logical consistency and computational efficiency. Evaluated on over 50,000 high-quality samples, our method significantly outperforms existing baselines across diverse instructions, achieving substantial improvements in generation fidelity and detail accuracy.

Abstract

Recent unified models integrate multimodal understanding and generation within a single framework. However, an "understanding-generation gap" persists, where models can capture user intent but often fail to translate this semantic knowledge into precise pixel-level manipulation. This gap results in two bottlenecks in anything-to-image task (X2I): the attention entanglement bottleneck, where blind planning struggles with complex prompts, and the visual refinement bottleneck, where unstructured feedback fails to correct imperfections efficiently. In this paper, we propose a novel framework that empowers unified models to autonomously switch between generation strategies based on instruction complexity and model capability. To achieve this, we construct a hierarchical data pipeline that constructs execution paths across three adaptive modes: direct generation for simple cases, self-reflection for quality refinement, and multi-step planning for decomposing complex scenarios. Building on this pipeline, we contribute a high-quality dataset with over 50,000 samples and implement a two-stage training strategy comprising SFT and RL. Specifically, we design step-wise reasoning rewards to ensure logical consistency and intra-group complexity penalty to prevent redundant computational overhead. Extensive experiments demonstrate that our method outperforms existing baselines on X2I, achieving superior generation fidelity among simple-to-complex instructions. The code is released at https://github.com/WeChatCV/Interleaved_Visual_Reasoner.

Problem

• understanding-generation gap
• attention entanglement bottleneck
• visual refinement bottleneck
• anything-to-image
• multimodal models

Innovation

• self-adaptive reasoning
• interleaved visual generation
• understanding-generation gap
• multi-step planning
• reinforcement learning for X2I