LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

2026-01-15

AI Summary

In multimodal implicit reasoning, lightweight student models often rely excessively on linguistic priors while neglecting genuine visual perception, leading to significant divergence in visual attention from their teacher counterparts. To address this, this work proposes a novel paradigm that aligns the "latent visual thinking" of student and teacher models. Specifically, it employs autoregressive reconstruction of the teacher’s visual semantics and attention trajectories to align their dynamic visual reasoning processes prior to text generation. A curriculum-based sensory gating mechanism is further introduced to suppress shortcut learning. This approach represents the first explicit modeling and transfer of the teacher’s dynamic visual attention, achieving up to a 16.9% performance gain on complex reasoning tasks and enabling a 3B-parameter model to surpass both larger open-source models and closed-source systems such as GPT-4o.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.

Problem

• Perception Gap
• multimodal reasoning
• visual grounding
• knowledge distillation
• latent reasoning

Innovation

• latent visual alignment
• visual grounding
• multimodal reasoning
• attention trajectory
• perception gap