Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions

2025-04-20

AI Summary

To address the escalating computational and data requirements posed by increasingly large language models (LLMs), this paper systematically investigates a synergistic compression paradigm integrating knowledge distillation (KD) and dataset distillation (DD), balancing inference capability preservation with linguistic diversity. We propose, for the first time, a tri-faceted framework unifying multi-teacher alignment, gradient-matching synthetic data generation, and latent-space regularization—overcoming inherent limitations of single-stage distillation in retaining emergent capabilities. Our approach incorporates task-aligned distillation, rationale-driven training, and generative dataset synthesis to enable lightweight deployment tailored to healthcare and education domains. Experiments demonstrate that the distilled models retain over 90% of original performance while substantially reducing parameter count and inference latency. This work establishes a novel pathway toward efficient, sustainable LLM deployment without compromising functional fidelity or generative richness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Data Compression

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles.

Problem

• Compressing Large Language Models efficiently while preserving capabilities
• Integrating Knowledge and Dataset Distillation for scalable strategies
• Addressing challenges in model scalability and emergent ability preservation

Innovation

• Knowledge Distillation with task-specific alignment
• Dataset Distillation via gradient matching
• Integrating KD and DD for scalable compression