CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs

2025-10-01

AI Summary

To address the high cost and low efficiency of in-context learning (ICL) and fine-tuning for multi-step reasoning tasks, this paper proposes CoT Vectors—a novel method that transfers and probes the internal reasoning mechanisms of large language models (LLMs) via compact vector representations. We first identify a systematic three-stage reasoning process common across LLMs and design task-agnostic, learnable CoT vectors trained lightly within a teacher–student framework to provide stable reasoning guidance. With only a minimal number of trainable parameters—far fewer than those required by parameter-efficient fine-tuning—CoT Vectors significantly outperform standard ICL across multiple reasoning benchmarks and approach the performance of state-of-the-art fine-tuning methods. Crucially, CoT vectors serve as structured probes that uncover latent organizational principles underlying LLM reasoning capabilities, establishing a new paradigm for efficient and interpretable reasoning enhancement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Chain-of-Thought (CoT) prompting has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing implementations, such as in-context learning and fine-tuning, remain costly and inefficient. To improve CoT reasoning at a lower cost, and inspired by the task vector paradigm, we introduce CoT Vectors, compact representations that encode task-general, multi-step reasoning knowledge. Through experiments with Extracted CoT Vectors, we observe pronounced layer-wise instability, manifesting as a U-shaped performance curve that reflects a systematic three-stage reasoning process in LLMs. To address this limitation, we propose Learnable CoT Vectors, optimized under a teacher-student framework to provide more stable and robust guidance. Extensive evaluations across diverse benchmarks and models demonstrate that CoT Vectors not only outperform existing baselines but also achieve performance comparable to parameter-efficient fine-tuning methods, while requiring fewer trainable parameters. Moreover, by treating CoT Vectors as a probe, we uncover how their effectiveness varies due to latent space structure, information density, acquisition mechanisms, and pre-training differences, offering new insights into the functional organization of multi-step reasoning in LLMs. The source code will be released.

Problem

• Improving Chain-of-Thought reasoning efficiency in LLMs
• Addressing layer-wise instability in reasoning mechanisms
• Probing latent space structure of multi-step reasoning

Innovation

• Introducing compact CoT Vectors for reasoning knowledge
• Optimizing Learnable CoT Vectors via teacher-student framework
• Using CoT Vectors as probes to analyze reasoning mechanisms