MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation

2026-05-02

AI Summary

Existing knowledge distillation methods for large language models typically align representations only at fixed layers or token levels, overlooking the dynamic evolution of semantic representations across model depth and thus failing to fully transfer the teacher model’s internal structural information. This work proposes a Multi-granularity Trajectory Alignment (MTA) framework that extends representation alignment from a static to a dynamic inter-layer trajectory perspective. Specifically, MTA employs token-level alignment in lower layers to preserve lexical details and introduces phrase-level alignment in higher layers to capture compositional semantics. It further incorporates a dynamic structural alignment loss to match the relative geometric relationships among semantic units across layers, complemented by hidden representation alignment. By integrating linguistic hierarchical structure with the depth-dependent abstraction property of Transformer representations, MTA significantly outperforms existing distillation approaches on standard benchmarks, with ablation studies confirming the contribution of each component.

Abstract

Knowledge distillation is a key technique for compressing large language models (LLMs), but most existing methods align representations at fixed layers or token-level outputs, ignoring how representations evolve across depth. As a result, the student is only weakly guided to capture the teacher's internal relational structure during distillation, which limits knowledge transfer. To address this limitation, we propose Multi-Granular Trajectory Alignment (MTA), a framework that aligns teacher and student representations along their layer-wise transformation trajectory. MTA adopts a layer-adaptive strategy: lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans (e.g., noun and verb phrases) to capture compositional semantics. We instantiate this idea through a Dynamic Structural Alignment loss that matches the relative geometry among semantic units within each layer. This design is motivated by empirical findings that Transformer representations become increasingly abstract with depth, and is also consistent with linguistic views in which higher-level meaning emerges through the composition of lower-level lexical units. We further incorporate a Hidden Representation Alignment loss to directly align selected teacher-student layers. Experiments show that MTA consistently outperforms state-of-the-art baselines on standard benchmarks, with ablations confirming the contribution of each component.

Problem

• knowledge distillation
• large language models
• representation alignment
• layer-wise trajectory
• semantic composition

Innovation

• Multi-Granular Trajectory Alignment
• Knowledge Distillation
• Layer-wise Alignment
• Dynamic Structural Alignment
• Compositional Semantics