Transformers Provably Learn to Internalize Chain-of-Thought

2026-05-27

AI Summary

This work addresses the high computational cost of explicit Chain-of-Thought (CoT) reasoning and the lack of theoretical guarantees in implicit CoT (ICoT). The authors propose Log-ICoT, a training strategy that geometrically reduces thought tokens and internalizes reasoning steps into the hidden states of a multi-layer Transformer. For the first time, this approach provides theoretical guarantees for ICoT, reducing the required number of training stages from linear to logarithmic and extending single-layer results to deep architectures. Theoretical analysis shows that Log-ICoT learns k-parity functions with polynomial sample complexity, matching the sample efficiency of explicit CoT. Experiments confirm that reasoning is progressively internalized across deeper layers while entirely eliminating additional inference-time overhead.

Abstract

Chain-of-Thought (CoT) prompting substantially improves the sample efficiency of transformers, reducing the complexity of tasks like parity learning from exponential to polynomial in the input length. However, generating explicit reasoning steps at inference is computationally expensive. Implicit Chain-of-Thought (ICoT) has emerged as a promising empirical remedy that trains models to internalize intermediate steps within their hidden states, but its theoretical foundations remain poorly understood. We give the first theoretical analysis of ICoT, proving that an $L$-layer transformer trained under our proposed Log-ICoT curriculum learns $k$-parity with $\mathsf{poly}(n)$ samples and $L = \log_2 k$ training stages. This matches the sample efficiency of explicit CoT while eliminating its inference overhead, and extends prior one-layer parity guarantees to multi-layer architectures. Compared to standard ICoT, which removes thinking tokens one at a time, Log-ICoT removes them in geometric chunks, reducing the number of stages from linear in $k$ to logarithmic. Experiments on multi-layer transformers confirm the theory and visualize how reasoning is progressively absorbed into deeper layers.

Problem

• Chain-of-Thought
• Implicit Reasoning
• Transformers
• Sample Efficiency
• Parity Learning

Innovation

• Implicit Chain-of-Thought
• Log-ICoT
• transformer theory
• sample efficiency
• internalized reasoning