To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

2026-07-12

AI Summary

This work addresses the susceptibility of current retrieval-augmented question answering systems to hallucination when retrieval fails, due to the absence of effective mitigation mechanisms. The authors propose an active abstention mechanism grounded in an abstention-aware reinforcement learning framework, which integrates query-level prior capability estimation and dynamic abstention rewards to encourage the model to refrain from answering under high uncertainty. The approach features continuous policy updates driven by outcome-based rewards and introduces a novel evaluation metric, RA-F1, designed to jointly optimize accuracy and reliability. Experimental results demonstrate that, compared to non-abstaining baselines, the method achieves up to a 10.3% absolute improvement in precision and a 2.9% gain in RA-F1, with only marginal degradation in original answer accuracy.

Abstract

Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.

Problem

• hallucinations
• search-agent
• reinforcement learning
• abstention
• open-domain QA

Innovation

• Abstention-Aware Reinforcement Learning
• Hallucination Mitigation
• Search-Agent Reliability
• RA-F1 Metric
• Large Language Models