🤖 AI Summary
This survey addresses the integration of information retrieval (IR) and natural language processing (NLP), focusing on systematic comparison across sparse, dense, and hybrid retrieval paradigms. Methodologically, it unifies classical and modern frameworks—including Lucene, Anserini, Pyserini—as well as foundational models (vector space, probabilistic, inference networks) and state-of-the-art pre-trained Transformers (e.g., BERT). Its primary contribution is the first comprehensive, cross-paradigm empirical evaluation of sparse, dense, and hybrid retrieval, extended to emerging application domains: cross-lingual IR, argument mining, privacy-preserving retrieval, and hate speech detection. The study clarifies co-evolutionary trajectories between IR and NLP, identifies core challenges in accuracy, scalability, and ethical robustness, and proposes future research directions toward trustworthy, fair, and efficient retrieval systems.
📝 Abstract
This review paper explores recent advancements and emerging approaches in Information Retrieval (IR) applied to Natural Language Processing (NLP). We examine traditional IR models such as Boolean, vector space, probabilistic, and inference network models, and highlight modern techniques including deep learning, reinforcement learning, and pretrained transformer models like BERT. We discuss key tools and libraries - Lucene, Anserini, and Pyserini - for efficient text indexing and search. A comparative analysis of sparse, dense, and hybrid retrieval methods is presented, along with applications in web search engines, cross-language IR, argument mining, private information retrieval, and hate speech detection. Finally, we identify open challenges and future research directions to enhance retrieval accuracy, scalability, and ethical considerations.