VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

2026-09-07

AI Summary

为了解决文本到图像系统中缺乏有效连接语言和视觉信号的中间表示问题，本文提出VoT框架，通过引入一个离散的视觉思考层来生成可解释和可控的图像。

Abstract

Current text-to-image systems typically employ a"text encoder plus diffusion decoder"paradigm, in which text semantics directly modulate continuous latent noise. Despite their success, these methods lack an explicit, interpretable intermediate representation that effectively bridges high-level linguistic semantics and low-level visual signals. In this paper, we propose Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs). Instead of treating VLMs merely as text encoders, we use them as multimodal planners that generate discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels. We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses. These objectives make the tokens semantically readable by the VLM while preserving the visual information needed for generation. Experimental results demonstrate that VoT improves semantic alignment and provides a structured interface for interpretable and controllable generation.

Problem

• text-to-image
• semantic alignment
• multimodal representation
• visual signals
• linguistic semantics

Innovation

• Vision-of-Thought
• discrete visual-thinking layer
• multimodal planners
• VoT tokenizer
• semantic alignment