MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora

2025-07-14

AI Summary

To address the high computational cost and frequent full-model retraining required for index updates in generative retrieval models under dynamic corpora, this paper proposes an incremental learning framework that avoids complete model retraining. The method integrates scalable LoRA with a Mixture-of-Experts (MoE) architecture and introduces a layer-wise out-of-distribution (OOD) document detection mechanism to guide sparse, targeted expert expansion. This enables sublinear parameter growth and efficient modular scaling, allowing model capabilities to evolve dynamically with incoming data while minimizing training overhead and parameter expansion. Evaluated on NQ320k and MS MARCO Passage, the approach outperforms full-model update baselines in retrieval effectiveness, reduces training time by over 60%, and adds fewer than 5% new parameters.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationally expensive and impractical under resource constraints. We propose MixLoRA-DSI, a novel framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution (OOD)-driven expansion strategy. Instead of allocating new experts for each new corpus, our proposed expansion strategy enables sublinear parameter growth by selectively introducing new experts only when significant number of OOD documents are detected. Experiments on NQ320k and MS MARCO Passage demonstrate that MixLoRA-DSI outperforms full-model update baselines, with minimal parameter overhead and substantially lower training costs.

Problem

• Efficiently update generative retrieval models for dynamic corpora
• Reduce computational cost of model retraining with new documents
• Minimize parameter growth while maintaining retrieval performance

Innovation

• Expandable mixture of Low-Rank Adaptation experts
• Layer-wise OOD-driven expansion strategy
• Sublinear parameter growth via selective expert introduction