Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning

2025-02-25

AI Summary

This work investigates efficient distillation of large language models’ (LLMs) chain-of-thought (CoT) reasoning capabilities into small language models (SLMs), balancing computational efficiency and reasoning performance. We conduct large-scale controlled experiments across seven mathematical and commonsense reasoning benchmarks, systematically varying four teacher LLMs, seven student architectures, CoT granularity levels, CoT formatting strategies, and teacher selection criteria. Key findings reveal: (i) a non-monotonic granularity effect in CoT distillation—neither finest nor coarsest granularities yield optimal performance; (ii) minimal impact of CoT formatting on student outcomes; and (iii) no positive correlation between teacher model strength and student performance, highlighting the need to balance teacher diversity and reasoning complexity. Based on these insights, we propose a student-adaptive CoT distillation strategy that significantly improves SLM generalization and stability across multi-task reasoning. All code and datasets are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.

Problem

• Identify factors for CoT distillation.
• Examine granularity, format, and teacher models.
• Optimize CoT strategies for small language models.

Innovation

• Distilling CoT into SLMs
• Examining granularity and format
• Tailoring CoT strategies specifically