Reproducing NevIR: Negation in Neural Information Retrieval

2025-02-19

AI Summary

This work addresses the long-overlooked challenge of negation understanding in neural information retrieval (IR), where language models (LMs) exhibit significant deficiencies. We propose a systematic evaluation framework and conduct a comprehensive assessment of mainstream IR models’ ranking performance on negated queries, using the NevIR and ExcluIR benchmarks. Our findings reveal that most models perform near-randomly; although listwise large language model (LLM) re-rankers achieve the best and most robust performance across both negation tasks, they still substantially underperform human-level capability. Moreover, distributional discrepancies between ExcluIR and NevIR hinder cross-dataset generalization. This study provides the first empirical evidence of the relative advantages—and fundamental limitations—of listwise LLM re-rankers in negation-aware retrieval, offering critical insights and concrete directions for advancing negation-sensitive IR modeling.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Negation is a fundamental aspect of human communication, yet it remains a challenge for Language Models (LMs) in Information Retrieval (IR). Despite the heavy reliance of modern neural IR systems on LMs, little attention has been given to their handling of negation. In this study, we reproduce and extend the findings of NevIR, a benchmark study that revealed most IR models perform at or below the level of random ranking when dealing with negation. We replicate NevIR's original experiments and evaluate newly developed state-of-the-art IR models. Our findings show that a recently emerging category - listwise Large Language Model (LLM) rerankers - outperforms other models but still underperforms human performance. Additionally, we leverage ExcluIR, a benchmark dataset designed for exclusionary queries with extensive negation, to assess the generalizability of negation understanding. Our findings suggest that fine-tuning on one dataset does not reliably improve performance on the other, indicating notable differences in their data distributions. Furthermore, we observe that only cross-encoders and listwise LLM rerankers achieve reasonable performance across both negation tasks.

Problem

• Challenges in handling negation in IR
• Performance gap between models and humans
• Limited generalizability of negation understanding

Innovation

• Listwise LLM rerankers enhance negation handling
• ExcluIR assesses negation understanding generalizability
• Cross-encoders improve performance across negation tasks