Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers

2024-11-07

arXiv.org

AI Summary

Learning-free sparse retrievers suffer from significantly lower relevance compared to dense or sparse dual-encoder models. Method: This paper proposes an IDF-aware FLOPS loss and a heterogeneous ensemble knowledge distillation framework. Contributions/Results: (1) IDF-weighted sparse regularization mitigates relevance degradation caused by sparsification; (2) heterogeneous knowledge distillation jointly leverages dense and sparse dual-encoder teachers, unifying multi-source supervision signals via consistent normalization; (3) the framework supports inverted-index-compatible sparse representation learning and score aggregation. On the BEIR benchmark, it achieves a +3.3 NDCG@10 gain over prior learning-free sparse retrievers, matches the relevance of the best dual-encoder sparse models, and incurs only 1.1× inference latency relative to BM25—marking the first learning-free sparse retriever to simultaneously achieve state-of-the-art accuracy and efficiency.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Learned sparse retrieval, which can efficiently perform retrieval through mature inverted-index engines, has garnered growing attention in recent years. Particularly, the inference-free sparse retrievers are attractive as they eliminate online model inference in the retrieval phase thereby avoids huge computational cost, offering reasonable throughput and latency. However, even the state-of-the-art (SOTA) inference-free sparse models lag far behind in terms of search relevance when compared to both sparse and dense siamese models. Towards competitive search relevance for inference-free sparse retrievers, we argue that they deserve dedicated training methods other than using same ones with siamese encoders. In this paper, we propose two different approaches for performance improvement. First, we introduce the IDF-aware FLOPS loss, which introduces Inverted Document Frequency (IDF) to the sparsification of representations. We find that it mitigates the negative impact of the FLOPS regularization on search relevance, allowing the model to achieve a better balance between accuracy and efficiency. Moreover, we propose a heterogeneous ensemble knowledge distillation framework that combines siamese dense and sparse retrievers to generate supervisory signals during the pre-training phase. The ensemble framework of dense and sparse retriever capitalizes on their strengths respectively, providing a strong upper bound for knowledge distillation. To concur the diverse feedback from heterogeneous supervisors, we normalize and then aggregate the outputs of the teacher models to eliminate score scale differences. On the BEIR benchmark, our model outperforms existing SOTA inference-free sparse model by extbf{3.3 NDCG@10 score}. It exhibits search relevance comparable to siamese sparse retrievers and client-side latency only extbf{1.1x that of BM25}.

Problem

• Improving search relevance for inference-free sparse retrievers
• Developing dedicated training methods for sparse retrievers
• Enhancing performance via IDF-aware penalty and ensemble distillation

Innovation

• IDF-aware penalty for matching function
• Heterogeneous ensemble knowledge distillation framework
• Normalize and aggregate teacher model outputs