What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis

2026-06-18

AI Summary

Implicit chain-of-thought reasoning relying solely on outcome supervision is prone to semantic drift and gradient vanishing, hindering robust inference. This work addresses these limitations by reframing process supervision through an information-theoretic lens, decoupling it into trajectory and spatial supervision. Rather than enforcing geometric compression, the proposed approach preserves informational fidelity in the reasoning space via mutual information maximization. The authors introduce a “dual collapse” mechanism to elucidate the root causes of failure and develop a dual-dimensional supervision framework that jointly governs trajectory and spatial aspects. Semantic structure is retained through generative reconstruction instead of rigid geometric constraints. Experiments demonstrate a strong correlation between reasoning accuracy and the fidelity of latent trajectory information, establishing an “information–performance binding” principle that offers principled guidance for supervising implicit reasoning systems.

Abstract

Latent Chain-of-Thought (CoT) internalizes reasoning within continuous hidden states, offering a promising alternative to verbose discrete reasoning traces. However, robust latent reasoning remains difficult because outcome supervision provides weak learning signals and leaves latent trajectories prone to semantic drift. In this work, we analyze Latent CoT from an information-theoretic perspective and identify this failure as a dual collapse: gradient attenuation along the optimization path and representational drift in the latent space. We further decompose process supervision into two complementary dimensions: Trajectory Supervision, which injects dense stepwise reasoning signals, and Space Supervision, which preserves the semantic structure of the latent manifold. Our analysis shows that rigid geometric compression can collapse the reasoning space, whereas generative reconstruction provides a more flexible semantic anchor that better preserves information capacity. To measure these effects, we introduce the Unified Latent Probe (ULP), which quantifies the mutual information between latent trajectories and explicit reasoning steps. Experiments reveal a clear Information-Performance Binding: reasoning accuracy depends on the information fidelity preserved in the latent chain. These findings provide a principled framework for latent reasoning supervision and suggest shifting from geometric imitation toward mutual information maximization. Our code is available at \href{https://github.com/EIT-NLP/Supervision-in-Latent-CoT}{this repository}.

Problem

• Latent Chain-of-Thought
• supervision
• semantic drift
• information fidelity
• reasoning

Innovation

• Latent Chain-of-Thought
• information-theoretic analysis
• process supervision
• mutual information
• semantic drift