Decoder-based Sense Knowledge Distillation

2026-02-25

AI Summary

Existing decoder-only large language models struggle to effectively leverage structured lexical semantic knowledge during text generation, resulting in insufficient inheritance of semantic relationships. This work proposes a lexical semantic knowledge distillation framework tailored for such models, which, for the first time, integrates structured semantic knowledge from lexical resources into the training process. By combining lexical knowledge distillation, contextual embedding alignment, and decoder fine-tuning, the approach enhances the model’s semantic expressiveness without requiring on-the-fly dictionary lookups during inference. Experimental results demonstrate that the method significantly improves the model’s ability to inherit lexical semantic relations across multiple benchmarks, achieving a favorable balance between semantic richness and training efficiency.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Semantic Web

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.

Problem

• knowledge distillation
• decoder-based LLMs
• lexical knowledge
• word senses
• structured semantics

Innovation

• knowledge distillation
• decoder-based LLMs
• lexical resources
• word sense
• structured semantics