CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

2026-06-30

AI Summary

This work addresses the limitations of traditional multimodal reasoning methods that rely on textual chain-of-thought (CoT), which suffer from slow inference and constraints imposed by language expression. The authors propose CoLT, a novel framework that enables efficient reasoning through a small number of implicit latent states, eliminating the need for verbose intermediate text generation. CoLT introduces a lightweight external decoder trained under dual supervision—forward (predicting the next step) and backward (aligning with context)—alongside a coherence constraint on internal latent states to ensure stable training and semantically meaningful reasoning chains. During inference, the supervisory modules are removed to maximize efficiency. Experiments demonstrate that CoLT outperforms existing implicit reasoning and image-augmented visual reasoning approaches across eight benchmarks, achieving a 10.1× reduction in overall reasoning time and a 22.6× decrease in text decoding latency compared to textual CoT.

Abstract

Chain-of-thought (CoT) reasoning has enabled multi-modal large language models (MLLMs) to tackle complex visual reasoning tasks by generating explicit intermediate reasoning steps in natural language. However, this text-based reasoning paradigm is inherently slow at inference time with even thousands of tokens and fundamentally constrained by the expressiveness of natural language. In this paper, we propose CoLT, (Chain of Latent Thoughts), a novel framework that teaches multi-modal models to reason through a chain of latent thought representations instead of verbose text tokens, which can perform thinking with as few as 3 steps. Naively forcing the model to think with latent states easily produces meaningless semantics and makes training unstable. To effectively regulate the latent reasoning process, we introduce a lightweight external decoder that provides step-level supervision for each latent reasoning step in two complementary directions: a forward mode that decodes latent thoughts into the textual reasoning of the next step, and a backward mode that aligns decoder hidden states with the model's latent thoughts given preceding textual context. We further incorporate internal supervision that encourages coherent step-by-step latent transitions. The decoder and internal supervision are removed during inference to maintain high efficiency of latent reasoning. Extensive experiments on eight benchmarks demonstrate that CoLT not only outperforms existing latent reasoning methods such as CODI and SIM-CoT, but also surpasses latent visual reasoning approaches that rely on auxiliary images with costly annotation requirements. Compared to text CoT methods, CoLT can notably reduce the inference time by 10.1$\times$ and text decoding time by 22.6$\times$. Code is released at https://github.com/hulianyuyy/CoLT.

Problem

• Chain-of-Thought
• multi-modal reasoning
• latent representations
• inference efficiency
• visual reasoning

Innovation

• Chain of Latent Thoughts
• latent reasoning
• multi-modal LLMs
• efficient inference
• step-level supervision