Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

2024-08-19

arXiv.org

AI Summary

To address insufficient user personalization modeling and negative transfer caused by uniform LoRA adaptation in sequential recommendation, this paper proposes iLoRA—a novel instance-level LoRA-MoE fusion framework tailored for sequential recommendation. The core innovation lies in reformulating sequential recommendation as a multi-task learning problem and designing a sequence-aware gating mechanism that dynamically assigns expert weights based on user behavioral sequences, enabling fine-grained, instance-driven parameter adaptation. Built upon parameter-efficient fine-tuning (PEFT), iLoRA introduces less than 1% additional trainable parameters. Extensive experiments on three benchmark datasets demonstrate an average 11.4% improvement in Hit Ratio, substantial mitigation of negative transfer, and a favorable balance between model efficiency and personalized recommendation accuracy.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches are eager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4% over basic LoRA in the hit ratio metric, with less than a 1% relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code are available at https://github.com/AkaliKong/iLoRA.

Problem

• LoRA
• Sequential Recommendation Systems
• Personalization

Innovation

• iLoRA
• Personalized Sequential Recommendation
• Expert Involvement