Towards Dynamic Dense Retrieval with Routing Strategy

2026-02-25

AI Summary

This work addresses the limitations of traditional dense retrieval, which requires full-model fine-tuning for new tasks and struggles to adapt efficiently to new domains under data scarcity while incurring high update costs. To overcome these challenges, the authors propose Dynamic Dense Retrieval (DDR), the first approach to integrate modular prefix tuning with a dynamic routing mechanism into dense retrieval. DDR constructs lightweight, domain-specific prefix modules and dynamically selects routing paths based on input queries, enabling effective cross-domain transfer with only 2% trainable parameters. Evaluated on six zero-shot downstream tasks, DDR substantially outperforms standard dense retrieval methods, achieving superior generalization performance while significantly reducing training overhead.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The \textit{de facto} paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed \textit{dynamic dense retrieval} (DDR). DDR uses \textit{prefix tuning} as a \textit{module} specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.

Problem

• dense retrieval
• domain adaptation
• model updating
• parameter efficiency
• zero-shot retrieval

Innovation

• dynamic dense retrieval
• prefix tuning
• routing strategy
• parameter-efficient adaptation
• zero-shot retrieval