When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR

2025-06-02

AI Summary

To address performance degradation of dense retrievers on dynamic corpora due to distributional shift, this paper formally introduces and formulates the novel task of *Retriever Out-of-Distribution (OOD) Corpus Prediction*, aiming to proactively identify whether newly arriving documents lie outside the retriever’s original training distribution prior to indexing. To this end, we propose GradNormIR—a fully unsupervised, zero-shot OOD detection method that leverages gradient norm statistics derived from backpropagated gradients of pre-trained retrievers on query–document pairs, requiring neither labels nor fine-tuning. Evaluated on the multi-domain BEIR benchmark, GradNormIR achieves high-precision identification of OOD documents, enabling informed, targeted index updates. Experimental results demonstrate that integrating GradNormIR into dynamic retrieval update strategies significantly enhances retrieval robustness and sustains long-term system effectiveness under evolving corpus distributions.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR, an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency.

Problem

• Detecting when dense retrievers need updates in evolving corpora
• Identifying out-of-distribution corpora before indexing documents
• Improving retrieval robustness using unsupervised gradient norm methods

Innovation

• Detects OOD corpora using gradient norms
• Unsupervised approach for timely updates
• Enhances retrieval robustness and efficiency