Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home

2025-01-22

AI Summary

This work addresses three key challenges in retrieval-augmented generation (RAG): (1) models’ inability to distinguish between internal knowledge and external retrieved information, (2) insufficient uncertainty modeling, and (3) noise from irrelevant retrievals. We systematically evaluate 35 adaptive retrieval and uncertainty estimation methods across six question-answering benchmarks. To our knowledge, this is the first comprehensive comparative study of these two paradigms. Results show that uncertainty estimation significantly outperforms LLM-intrinsic-knowledge-based adaptive retrieval—by over 40% in computational efficiency, with superior self-knowledge modeling and more reliable delineation of knowledge boundaries. Using ten multidimensional metrics, we demonstrate that uncertainty-aware methods maintain answer accuracy while substantially improving reasoning efficiency and interpretability. Our findings challenge prevailing RAG adaptation assumptions and broaden the design space for adaptive mechanisms.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but they often neglect efficiency evaluations and comparisons with uncertainty estimation techniques. We bridge this gap by conducting a comprehensive analysis of 35 adaptive retrieval methods, including 8 recent approaches and 27 uncertainty estimation techniques, across 6 datasets using 10 metrics for QA performance, self-knowledge, and efficiency. Our findings show that uncertainty estimation techniques often outperform complex pipelines in terms of efficiency and self-knowledge, while maintaining comparable QA performance.

Problem

• Efficiency
• Uncertainty Handling
• Relevance Filtering

Innovation

• Uncertainty Estimation Techniques
• Knowledge Integration
• Efficiency Optimization