Bidirectional Knowledge Distillation for Enhancing Sequential Recommendation with Large Language Models

2025-05-23

AI Summary

To address the low collaboration efficiency, static unidirectional knowledge transfer, and high inference overhead when integrating large language models (LLMs) with conventional recommendation models (CRMs) in sequential recommendation, this paper proposes LLMD4Rec—a dynamic bidirectional distillation framework. It introduces the first parameter-free iterative bidirectional knowledge distillation mechanism, leveraging sample-adaptive weighting and output distribution alignment to enable mutual reinforcement between LLMs’ semantic understanding capabilities and CRMs’ collaborative signals. Evaluated on multiple real-world datasets, LLMD4Rec achieves significant accuracy gains (e.g., +12.7% average Recall@20) without increasing inference latency, balancing performance improvement and deployment practicality. Its core contribution lies in establishing the first dynamic bidirectional distillation paradigm between LLMs and CRMs—overcoming the limitations of unidirectional, static transfer—and paving a new path toward lightweight, efficient LLM-enhanced recommendation systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated exceptional performance in understanding and generating semantic patterns, making them promising candidates for sequential recommendation tasks. However, when combined with conventional recommendation models (CRMs), LLMs often face challenges related to high inference costs and static knowledge transfer methods. In this paper, we propose a novel mutual distillation framework, LLMD4Rec, that fosters dynamic and bidirectional knowledge exchange between LLM-centric and CRM-based recommendation systems. Unlike traditional unidirectional distillation methods, LLMD4Rec enables iterative optimization by alternately refining both models, enhancing the semantic understanding of CRMs and enriching LLMs with collaborative signals from user-item interactions. By leveraging sample-wise adaptive weighting and aligning output distributions, our approach eliminates the need for additional parameters while ensuring effective knowledge transfer. Extensive experiments on real-world datasets demonstrate that LLMD4Rec significantly improves recommendation accuracy across multiple benchmarks without increasing inference costs. This method provides a scalable and efficient solution for combining the strengths of both LLMs and CRMs in sequential recommendation systems.

Problem

• High inference costs in LLM-based sequential recommendation systems
• Static knowledge transfer between LLMs and conventional recommendation models
• Lack of dynamic bidirectional knowledge exchange in recommendation systems

Innovation

• Bidirectional knowledge distillation between LLMs and CRMs
• Sample-wise adaptive weighting for effective transfer
• Iterative optimization without additional parameters