SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA

2025-03-23

AI Summary

In complex question answering, first-stage retrieval suffers from low recall, severely constraining subsequent LLM-based reasoning due to limited context windows—especially under strict depth constraints (k ≤ 10), where performance degrades significantly. To address this, we propose a semantic-uncertainty-driven, neighborhood-aware retrieval framework: (1) an LLM generates answer candidates and quantifies their semantic uncertainty as feedback; (2) a dynamic document neighborhood graph is constructed and iteratively refined to optimize retrieval paths; and (3) retrieval and reasoning signals are tightly coupled in a closed-loop manner. This work is the first to integrate LLM-derived semantic uncertainty into retrieval feedback, enabling lightweight, high-recall retrieval under stringent depth limits. Evaluated on two complex QA benchmarks, our method achieves up to a 31.84% absolute improvement in overall performance, substantially outperforming state-of-the-art approaches.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Complex question-answering (QA) systems face significant challenges in retrieving and reasoning over information that addresses multi-faceted queries. While large language models (LLMs) have advanced the reasoning capabilities of these systems, the bounded-recall problem persists, where procuring all relevant documents in first-stage retrieval remains a challenge. Missing pertinent documents at this stage leads to performance degradation that cannot be remedied in later stages, especially given the limited context windows of LLMs which necessitate high recall at smaller retrieval depths. In this paper, we introduce SUNAR, a novel approach that leverages LLMs to guide a Neighborhood Aware Retrieval process. SUNAR iteratively explores a neighborhood graph of documents, dynamically promoting or penalizing documents based on uncertainty estimates from interim LLM-generated answer candidates. We validate our approach through extensive experiments on two complex QA datasets. Our results show that SUNAR significantly outperforms existing retrieve-and-reason baselines, achieving up to a 31.84% improvement in performance over existing state-of-the-art methods for complex QA.

Problem

• Improving recall in first-stage retrieval for complex QA
• Addressing bounded-recall problem with semantic uncertainty
• Enhancing retrieval accuracy using neighborhood-aware document exploration

Innovation

• Uses LLMs for Neighborhood Aware Retrieval
• Dynamically adjusts document relevance via uncertainty
• Iteratively explores document graph for better recall