Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens

2026-02-10

AI Summary

This work proposes Latent Thoughts Tuning, a novel framework that addresses feature collapse and training instability in existing continuous latent-space reasoning methods, which often suffer from redundant hidden state reuse or reliance on auxiliary models. The approach innovatively integrates lexical semantic signals into latent state construction and employs a Context-Prediction-Fusion mechanism to effectively combine contextual and predictive information. Furthermore, it leverages a three-stage curriculum learning strategy to enable dynamic switching between implicit and explicit reasoning modes. This design mitigates distributional mismatch and alignment issues, yielding significant improvements in both accuracy and robustness across multiple benchmarks, outperforming current implicit reasoning approaches.

Technology Category

• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While explicit Chain-of-Thought (CoT) equips Large Language Models (LLMs) with strong reasoning capabilities, it requires models to verbalize every intermediate step in text tokens, constraining the model thoughts to the discrete vocabulary space. Recently, reasoning in continuous latent space has emerged as a promising alternative, enabling more robust inference and flexible computation beyond discrete token constraints. However, current latent paradigms often suffer from feature collapse and instability, stemming from distribution mismatches when recurrently using hidden states as the input embeddings, or alignment issues when relying on assistant models. To address this, we propose Latent Thoughts Tuning (LT-Tuning), a framework that redefines how latent thoughts are constructed and deployed. Instead of relying solely on raw hidden states, our method introduces a Context-Prediction-Fusion mechanism that jointly leveraging contextual hidden states and predictive semantic guidance from the vocabulary embedding space. Combined with a progressive three-stage curriculum learning pipeline, LT-Tuning also enables dynamically switching between latent and explicit thinking modes. Experiments demonstrate that our method outperforms existing latent reasoning baselines, effectively mitigating feature collapse and achieving robust reasoning accuracy.

Problem

• latent reasoning
• feature collapse
• distribution mismatch
• alignment issues
• continuous latent space

Innovation

• Latent Thoughts Tuning
• Context-Prediction-Fusion
• latent reasoning
• feature collapse mitigation
• curriculum learning