Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model

2026-06-11

AI Summary

This work addresses the high computational cost and cross-modal interference inherent in existing vision-language reinforcement learning approaches based on autoregressive models, which require full image regeneration and employ shared reward mechanisms. To overcome these limitations, we introduce, for the first time, a multimodal discrete diffusion model into this task, enabling efficient inference through localized visual editing. We further propose a decoupled reward allocation strategy that assigns separate rewards to textual and visual segments. Integrated with the GRPO algorithm, our method achieves substantial computational savings—reducing inference FLOPs by 26.9% compared to autoregressive baselines—while significantly improving performance: the decoupled reward scheme yields an 11.2% gain over joint reward assignment and a 38.04% improvement over the base model, all without compromising task effectiveness.

Abstract

RL-based post-training has been widely adopted to enable interleaved visual and textual reasoning in unified multimodal models capable of both text and image generation. However, most existing approaches are built upon autoregressive (AR) unified models, which require full image regeneration during visual reasoning. In this work, we demonstrate that multimodal discrete diffusion models are effective alternatives to AR models for reinforcement learning in interleaved reasoning, owing to their ability to perform efficient visual rollouts via localized visual editing rather than full image-token regeneration. This reduces rollout computation during GRPO by 26.9\% compared to AR baselines, with minimal performance drop. Despite the improved efficiency, we find that joint reward assignment, which employs a shared reward signal across modalities, introduces cross-modal interference between unrelated image and text token sequences during RL updates. To address this issue, we propose factorized reward assignment, a strategy that assigns rewards independently to text and vision segments. With factorized reward assignment, our RL approach achieves an 11.2% improvement over joint reward assignment and a 38.04% improvement over the base model.

Problem

• reinforcement learning
• multimodal reasoning
• discrete diffusion model
• reward assignment
• visual-textual thinking

Innovation

• discrete diffusion model
• reinforcement learning
• multimodal reasoning
• factorized reward assignment
• visual-textual interleaving