Distillation Quantification for Large Language Models

2025-01-22

AI Summary

This work addresses the critical issue of model homogenization in large language model (LLM) knowledge distillation, which undermines small models’ generalization capability and identity uniqueness. We propose the first systematic framework for quantifying “distillation degree”—a novel metric capturing the extent to which distilled models lose their intrinsic identity and converge toward uniform behavior. Methodologically, we introduce a multi-granularity distillation-degree paradigm integrating semantic consistency checking, cross-model response embedding similarity computation, and identity-attribute deviation analysis—jointly modeling identity-cognition conflicts and response homogenization. Empirical results reveal significantly lower distillation degrees in closed-source models (Claude, Gemini, Doubao); base models exhibit greater susceptibility to homogenization than aligned models; and all code and datasets are publicly released. This work establishes a reproducible, principled evaluation benchmark for controllable knowledge transfer and model diversity preservation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Model distillation is a technique for transferring knowledge from large language models (LLMs) to smaller ones, aiming to create resource-efficient yet high-performing models. However, excessive distillation can lead to homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification.

Problem

• Knowledge Distillation
• Language Models
• Efficiency and Uniqueness

Innovation

• Efficiency-Uniqueness Balance
• Identity Information Consistency
• Model Response Similarity