Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations

2026-06-15

AI Summary

This work addresses the lack of interpretable visual intermediate representations in existing multimodal large language models, which typically rely solely on textual chain-of-thought (CoT) reasoning. The authors propose Gen-VCoT, a novel framework that, for the first time, leverages diffusion-generated RGB images as interpretable visual CoT intermediates. It introduces a three-stage multimodal reasoning paradigm encompassing visual grounding, geometric reasoning, and semantic reasoning, augmented with an adaptive routing mechanism that dynamically selects reasoning depth. Integrating SAM segmentation, Marigold depth estimation, and Qwen2-VL, the method achieves performance gains of 25% and 50% on spatial and depth-related tasks, respectively. Although it shows slight degradation on simple factual questions and underperforms textual CoT on CLEVR (62.5% vs. 91.2%), this approach establishes a promising new direction for interpretable multimodal reasoning.

Abstract

Multimodal large language models (MLLMs) excel at visual reasoning but rely on text-based chain-of-thought (CoT), lacking interpretable visual intermediates. Existing methods use opaque tokens or external tools, missing key properties. We propose Gen-VCoT, a framework using expert vision models to generate RGB images as reasoning intermediates. It has three stages: visual grounding (SAM segmentation), geometric reasoning (Marigold depth maps), and semantic reasoning (Qwen2-VL integration). An adaptive router selects reasoning depth. Evaluations show Gen-VCoT improves spatial (25% better) and depth (50% better) questions, but may hurt simple factual queries. Text CoT outperforms visual intermediates on CLEVR (91.2% vs 62.5%), showing task-dependent optimal representations. Gen-VCoT establishes a new paradigm for interpretable multimodal reasoning.

Problem

• multimodal reasoning
• visual chain-of-thought
• interpretable intermediates
• RGB representations
• visual reasoning

Innovation

• Generative Visual Chain-of-Thought
• Diffusion-based RGB Intermediates
• Interpretable Multimodal Reasoning
• Adaptive Router
• Vision-Language Integration