Leveraging Latent Visual Reasoning in Silence

2026-05-18

AI Summary

This study investigates the necessity of implicit visual tokens during inference in latent visual reasoning and finds that their presence is not essential. To address this, the authors propose an attention-based reinforcement learning reward mechanism that, during training, guides effective interaction between implicit visual tokens and text generation, thereby internalizing visual information into the language model while preserving the flexibility of purely textual reasoning. The work redefines the value of latent visual reasoning in terms of “learned guidance efficacy” rather than “token presence at inference time.” This approach achieves consistent performance gains across multiple visual and perceptual reasoning benchmarks: even when implicit tokens are rarely explicitly generated during post-training inference, it significantly enhances visual grounding and textual reasoning accuracy.

Abstract

Latent visual reasoning involves visual evidence more directly in multimodal reasoning by inserting continuous latent tokens before textual generation. However, the necessity of these latent tokens at inference remains ambiguous. We show that replacing latent tokens with random noise or removing them completely causes little performance degradation across spatial reasoning benchmarks. Reinforcement learning further diminishes the latent generation behavior after post-training. These observations raise a central question: Is latent visual reasoning still meaningful? We argue that its value should be measured by how effectively latent tokens guide learning, rather than whether they persist as an inference-time format. Our analysis shows that latent reasoning is unevenly favorable across question types, yet hard task-level routing for applying latent generation is brittle. Motivated by these findings, we propose an attention-based reward that encourages generated latent tokens to interact with later text tokens during RL. This reward promotes latent utilization when the latent mode is activated while preserving the flexibility to use pure-text reasoning. Experiments show that our method improves performance across perception and visual reasoning benchmarks, even when latent tokens are rarely generated after post-training. Our results highlight that, without explicit expression at inference, latent visual reasoning can shape better visual grounding and more accurate textual reasoning in silence. Our code and trained models are publicly available at \href{https://github.com/ddydyd32/silent-lvr/tree/master}{GitHub} and \href{https://huggingface.co/collections/cornuHGF/silent-lvr}{Hugging Face}.

Problem

• latent visual reasoning
• multimodal reasoning
• inference-time necessity
• task-level routing
• visual grounding

Innovation

• latent visual reasoning
• reinforcement learning
• attention-based reward
• multimodal reasoning
• visual grounding