Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

2026-07-04

AI Summary

Existing unified multimodal models struggle to enable end-to-end joint optimization of interleaved text-image reasoning trajectories via reinforcement learning, primarily because image generation typically relies on supervised proxies that impede policy gradient propagation. This work proposes BRAID, a framework that formalizes multi-turn text-image-text interactions as a unified Markov decision process (MDP) for the first time and introduces a vision-language model as a judge to provide dense intermediate rewards, thereby enabling end-to-end cross-modal policy gradient optimization. By integrating modality-native policy gradients with denoising path optimization in diffusion models, BRAID substantially outperforms current baselines on spatial reasoning and visual perception benchmarks, demonstrating the efficacy of unified MDP modeling and vision-guided reasoning.

Abstract

Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge. Existing approaches apply RL exclusively to text steps, relegating image generation to supervised surrogates, preventing policy gradients from propagating through the full interleaved trajectory across heterogeneous modalities. This leaves the potential of RL for UMMs largely untapped. In the paper, we introduce \textbf{BRAID} (\textbf{B}ridging inte\textbf{R}le\textbf{A}ved mult\textbf{I}-modal reasoning as a unified \textbf{D}ecision process), a simple framework that casts multi-turn text-image-text reasoning as a unified Markov decision process (MDP), enabling joint optimization of textual and visual generation via a single, principled RL objective. BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths, each optimized through its modality-native policy gradient mechanism. To further address long-horizon credit assignment, BRAID employs a vision-language model (VLM) judge that scores each intermediate image on its reasoning utility, supplying dense turn-level feedback to sharpen learning at critical visual branches. Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines, confirming that a unified MDP formulation with vision-thinking guidance is essential for effective multi-modal reasoning.

Problem

• interleaved multi-modal reasoning
• reinforcement learning
• unified decision process
• policy gradient
• multi-turn generation

Innovation

• Unified Multi-Modal Reasoning
• Reinforcement Learning
• Markov Decision Process
• Interleaved Text-Image Generation
• Vision-Language Model