Exploring $ell_0$ Sparsification for Inference-free Sparse Retrievers

2025-04-21

AI Summary

FLOPS regularization underperforms in non-symmetric sparse retrieval (where queries are not encoded), limiting inference efficiency. Method: This paper introduces, for the first time, an end-to-end trainable sparsification framework based on the $ell_0$ norm into this paradigm. It proposes an $ell_0$-relaxed approximation, a differentiable hard-thresholding operator, and a two-stage sparse training strategy—eliminating hand-crafted rules and enabling truly inference-free query processing with zero parameters and zero computation on the query side. Contribution/Results: Evaluated on the BEIR cross-domain benchmark, our method achieves state-of-the-art performance among inference-free sparse retrievers, matching mainstream Siamese sparse models in effectiveness while significantly reducing online latency and computational resource overhead.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

With increasing demands for efficiency, information retrieval has developed a branch of sparse retrieval, further advancing towards inference-free retrieval where the documents are encoded during indexing time and there is no model-inference for queries. Existing sparse retrieval models rely on FLOPS regularization for sparsification, while this mechanism was originally designed for Siamese encoders, it is considered to be suboptimal in inference-free scenarios which is asymmetric. Previous attempts to adapt FLOPS for inference-free scenarios have been limited to rule-based methods, leaving the potential of sparsification approaches for inference-free retrieval models largely unexplored. In this paper, we explore $ell_0$ inspired sparsification manner for inference-free retrievers. Through comprehensive out-of-domain evaluation on the BEIR benchmark, our method achieves state-of-the-art performance among inference-free sparse retrieval models and is comparable to leading Siamese sparse retrieval models. Furthermore, we provide insights into the trade-off between retrieval effectiveness and computational efficiency, demonstrating practical value for real-world applications.

Problem

• Optimize sparsification for inference-free sparse retrieval models
• Address suboptimal FLOPS regularization in asymmetric retrieval scenarios
• Explore ℓ0 sparsification to enhance efficiency and effectiveness

Innovation

• Uses $ ell_0$ sparsification for inference-free retrieval
• Achieves state-of-the-art BEIR benchmark performance
• Balances retrieval effectiveness and computational efficiency