Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with constraints

2025-07-22

AI Summary

To address the misalignment between confidence and correctness in large language models (LLMs) for open-domain question answering, this paper proposes a multi-step reflective framework integrating confidence calibration with retrieval augmentation. Methodologically, it introduces soft reliability constraints into reinforcement learning for end-to-end optimization on Wikipedia data, jointly modeling evidence retrieval, multi-step reasoning, and dynamic confidence calibration. The core contribution lies in differentiable confidence modeling coupled with a retrieval-feedback loop, explicitly aligning model confidence with answer correctness. Experiments demonstrate substantial improvements across multiple open-domain QA benchmarks: expected calibration error (ECE) decreases by 32.7%, and accuracy-confidence correlation (AUC) increases by 18.4%. The approach maintains high recall while significantly enhancing output reliability, establishing a novel paradigm for deploying trustworthy LLMs.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Question Answering
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Improving the reliability of large language models (LLMs) is critical for deploying them in real-world scenarios. In this paper, we propose extbf{Deliberative Searcher}, the first framework to integrate certainty calibration with retrieval-based search for open-domain question answering. The agent performs multi-step reflection and verification over Wikipedia data and is trained with a reinforcement learning algorithm that optimizes for accuracy under a soft reliability constraint. Empirical results show that proposed method improves alignment between model confidence and correctness, leading to more trustworthy outputs. This paper will be continuously updated.

Problem

• Enhancing LLM reliability via reinforcement learning constraints
• Integrating certainty calibration with retrieval-based question answering
• Improving alignment between model confidence and correctness

Innovation

• Reinforcement learning optimizes accuracy under constraints
• Multi-step reflection verifies Wikipedia data
• Certainty calibration integrates with retrieval-based search