Unveiling the Mechanisms of Explicit CoT Training: How Chain-of-Thought Enhances Reasoning Generalization

2025-02-07

AI Summary

This work investigates the mechanistic underpinnings of explicit chain-of-thought (CoT) training in enhancing the reasoning generalization capabilities of large language models—specifically addressing (i) the advantages of CoT training and (ii) its intrinsic mechanisms. Method: We design controlled data distributions and a two-hop factual reasoning task, integrating circuit analysis with quantitative generalization evaluation. Contribution/Results: We first demonstrate that CoT training simultaneously improves both in-distribution (ID) and out-of-distribution (OOD) generalization while accelerating convergence. Second, models acquire systematic reasoning abilities even from noisy CoT demonstrations. Third, generalization unfolds via a multi-stage circuit evolution strictly aligned with training steps. Finally, we identify the data ratio λ and pattern structure as critical levers governing generalization behavior. These findings provide interpretable, quantifiable empirical evidence for the mechanistic foundations and generalization boundaries of CoT training.

Technology Category

• Machine Learning: Causal Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Training large language models (LLMs) with high-quality Chain-of-Thought (CoT) annotations has become a widely adopted strategy due to its significant enhancement of reasoning capabilities. To fully comprehend this approach, two questions naturally arise: (Q1) What advantages does training with CoT offer compared to training without CoT? (Q2) If there are advantages, what are the underlying mechanisms of explicit CoT training? Analyzing the advantages and mechanisms of CoT training is challenging due to the many factors involved. To address this, we conduct a detailed analysis using clear and controllable data distributions and, for the first time, reveal that CoT training offers the following advantages: (1) Training with CoT markedly improves reasoning generalization, extending it from in-distribution (ID) to both ID and out-of-distribution (OOD) scenarios, while also speeding up convergence; (2) Even when training with CoT includes a certain range of erroneous reasoning steps, it still enables the model to learn reasoning patterns, leading to systematic generalization. We further explore the underlying mechanisms from a circuit perspective: (1) The data distribution (e.g., ratio $lambda$ and pattern) plays a crucial role in influencing the model's systematic generalization; (2) CoT training (with two-hop facts) internalizes reasoning into a two-stage generalizing circuit, where the number of stages corresponds to the explicit reasoning steps during training. Our findings elucidate the mechanisms underlying explicit CoT training and offer critical insights into tuning strategies for LLMs to achieve robust generalization.

Problem

• Enhancing reasoning generalization in LLMs
• Understanding CoT training mechanisms
• Improving systematic generalization with CoT

Innovation

• Chain-of-Thought enhances reasoning generalization
• CoT training internalizes two-stage generalizing circuit
• Data distribution influences systematic generalization