Latent Visual States for Efficient Multimodal Reasoning

2026-06-23

AI Summary

Current large models rely on discrete tool calls for visual reasoning, resulting in rigid pipelines and high latency. This work proposes the EVA framework, which introduces continuous latent visual states—termed Latent Slot Tokens—as differentiable intermediate visual thoughts, replacing discrete invocations and enabling end-to-end joint training with textual tokens. To address policy shift during training, the authors devise a decoupled optimization strategy, D-GSPO, and construct EVA-230K, a novel image-text interleaved chain-of-thought dataset. Experimental results demonstrate that EVA significantly improves both reasoning performance and efficiency across multiple benchmarks.

Abstract

The integration of visual evidence has significantly enhanced the capabilities of large multimodal models. However, this integration predominantly relies on generating discrete outputs (etc., code or box coordinates) to invoke external tools, a process that introduces rigid dependencies and substantial latency. To overcome these limitations, we propose {EVA} (LatEnt Visual StAtes), a novel framework that natively generates continuous latent visual representations. These internal representations manifest as an adaptive sequence of Latent\_slot tokens, serving as intermediate visual thoughts during the reasoning process. These Latent\_slot tokens are then trained end-to-end with the discrete text tokens. This co-optimization, notably, causes extreme policy deviation in the 'transition window' following the Latent\_slot tokens. We develop D-GSPO (Decouple-GSPO) to target this root cause by decoupling the optimization of latent and discrete components. To support SFT, we construct EVA-230K, a high-quality text-image interleaved CoT dataset encompassing a diverse range of real-world scenes, documents, charts and OCR tasks. Extensive experiments across multiple benchmarks confirm that EVA achieves significant performance gains while enhancing inference efficiency.

Problem

• multimodal reasoning
• visual integration
• discrete outputs
• latency
• external tool dependency

Innovation

• Latent Visual States
• Multimodal Reasoning
• End-to-End Training
• Decoupled Optimization
• Interleaved CoT Dataset