FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking

2025-02-09

AI Summary

Real-world fact-checking critically requires retrieval of indirect network evidence—evidence that supports or refutes complex claims through non-direct, multi-hop reasoning—but existing methods underperform on bidirectional (supporting/contradicting) and indirect evidence retrieval, and lack zero-shot, open-domain evaluation benchmarks grounded in authentic production logs. To address this, we introduce FactIR: the first zero-shot, open-domain, bidirectional indirect evidence retrieval benchmark for fact-checking, constructed from real-world verification logs in Factiverse and enriched with multi-dimensional human annotations. We systematically evaluate state-of-the-art dense and sparse retrieval models on FactIR, revealing substantial deficiencies in retrieving contradictory and indirect evidence. FactIR fills a critical gap in open-domain fact-checking retrieval evaluation, providing a reproducible benchmark and identifying key directions for model improvement.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

The field of automated fact-checking increasingly depends on retrieving web-based evidence to determine the veracity of claims in real-world scenarios. A significant challenge in this process is not only retrieving relevant information, but also identifying evidence that can both support and refute complex claims. Traditional retrieval methods may return documents that directly address claims or lean toward supporting them, but often struggle with more complex claims requiring indirect reasoning. While some existing benchmarks and methods target retrieval for fact-checking, a comprehensive real-world open-domain benchmark has been lacking. In this paper, we present a real-world retrieval benchmark FactIR, derived from Factiverse production logs, enhanced with human annotations. We rigorously evaluate state-of-the-art retrieval models in a zero-shot setup on FactIR and offer insights for developing practical retrieval systems for fact-checking. Code and data are available at https://github.com/factiverse/factIR.

Problem

• Develops a real-world retrieval benchmark
• Enhances fact-checking with human annotations
• Evaluates zero-shot retrieval models effectiveness

Innovation

• Zero-shot retrieval benchmark
• Enhanced human annotations
• Real-world open-domain setup