MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation

2026-01-07

arXiv.org

AI Summary

This work addresses the limitations of small language models in emulating large language model reasoning, which often suffer from constrained in-domain performance and poor out-of-domain generalization due to passively copying a single reasoning trajectory and being susceptible to distribution shifts in the teacher model. To overcome these issues, the authors propose the MIND framework, which introduces an auxiliary “teaching assistant” network to generate multi-perspective chains of thought and adaptively aligns them with the student model’s current capabilities, thereby transforming knowledge distillation from passive imitation into active cognitive construction. MIND employs a feedback-driven inertial calibration mechanism that dynamically matches supervision signals to the student’s learning capacity, effectively mitigating catastrophic forgetting and fostering internalization of reasoning skills. Experiments demonstrate that MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks, with latent space analyses further confirming the successful internalization of reasoning capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While Large Language Models (LLMs) have emerged with remarkable capabilities in complex tasks through Chain-of-Thought reasoning, practical resource constraints have sparked interest in transferring these abilities to smaller models. However, achieving both domain performance and cross-domain generalization remains challenging. Existing approaches typically restrict students to following a single golden rationale and treat different reasoning paths independently. Due to distinct inductive biases and intrinsic preferences, alongside the student's evolving capacity and reasoning preferences during training, a teacher's"optimal"rationale could act as out-of-distribution noise. This misalignment leads to a degeneration of the student's latent reasoning distribution, causing suboptimal performance. To bridge this gap, we propose MIND, a capability-adaptive framework that transitions distillation from passive mimicry to active cognitive construction. We synthesize diverse teacher perspectives through a novel"Teaching Assistant"network. By employing a Feedback-Driven Inertia Calibration mechanism, this network utilizes inertia-filtered training loss to align supervision with the student's current adaptability, effectively enhancing performance while mitigating catastrophic forgetting. Extensive experiments demonstrate that MIND achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks, and our sophisticated latent space analysis further confirms the mechanism of reasoning ability internalization.

Problem

• knowledge distillation
• chain-of-thought reasoning
• cross-domain generalization
• reasoning alignment
• large language models

Innovation

• capability-aware distillation
• multi-perspective CoT
• active reasoning
• inertia calibration
• reasoning internalization