Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching

2025-07-07

AI Summary

This work addresses the limitation of conventional knowledge distillation (LLM→SLM) in text matching, which overlooks the domain expertise embedded in small language models (SLMs). We propose a **reverse knowledge distillation paradigm**, wherein large language models (LLMs) learn from SLMs. Methodologically, we reconfigure the LLM into an encoder-decoder architecture via LoRA and introduce **margin-aware contrastive learning**, aligning cross-architectural representations using fine-grained similarity signals generated by the SLM. To our knowledge, this is the first successful realization of SLM→LLM knowledge transfer, effectively bridging the modeling gap between encoder-only and decoder-based architectures. Empirical evaluation across financial, medical, and other domain-specific benchmarks—as well as real-world deployment—demonstrates substantial improvements in LLM matching performance. The proposed model has been fully deployed and operates stably in production.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Knowledge distillation typically involves transferring knowledge from a Large Language Model (LLM) to a Smaller Language Model (SLM). However, in tasks such as text matching, fine-tuned smaller models often yield more effective domain-specific representations, as they focus on optimizing the similarity of input pairs. To leverage both the specialized strengths of small models and the rich semantic understanding of LLMs, we introduce a flipped knowledge distillation paradigm, where LLM learns from SLM. Specifically, we address the architectural gap between decoder-only LLMs and smaller encoder-based models by reinterpreting LLMs in an encoder-decoder manner using LoRA. The encoder generates compressed representations, while the decoder maps them to the output space. During training, the encoder produces representations and their similarities, which are then aligned with the similarity scores produced by the teacher, using our proposed Margin-aware Contrastive Learning (MCL) approach. The MCL ensures accurate similarity for both positive and negative pairs, and adaptively handles the internal differences within positive and negative samples. Our paradigm requires only a reasonably good-performing SLM, allowing the LLM to achieve improved performance. Experiments on financial and healthcare benchmarks, as well as real-world applications, confirm its effectiveness, and the model has been fully deployed in an online environment.

Problem

• Enhance LLMs using small models' expertise in text matching
• Bridge architectural gap between LLMs and small encoder models
• Improve similarity accuracy with Margin-aware Contrastive Learning

Innovation

• Flipped knowledge distillation from SLM to LLM
• Encoder-decoder LLM reinterpretation using LoRA
• Margin-aware Contrastive Learning for similarity alignment