GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence

2026-01-08

arXiv.org

AI Summary

This work addresses the tendency of existing retrieval-augmented generation (RAG) systems to either hallucinate or erroneously abstain from answering when evidence is insufficient. To this end, the authors propose GRACE, a novel framework that unifies evidence-driven answer generation and reliable abstention within a reinforcement learning paradigm. GRACE employs a multi-stage gating reward function to guide the model in assessing evidence sufficiency and autonomously deciding whether to respond. Furthermore, it leverages heterogeneous retrievers to automatically generate diverse training samples, substantially reducing reliance on human annotations. Experimental results demonstrate that GRACE achieves state-of-the-art overall accuracy on two benchmarks, effectively balancing response correctness with justified abstention, while requiring only 10% of the labeled data used by prior methods.

Technology Category

• Machine Learning: Active Learning
• Knowledge Representation and Reasoning: Argumentation
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval-Augmented Generation (RAG) integrates external knowledge to enhance Large Language Models (LLMs), yet systems remain susceptible to two critical flaws: providing correct answers without explicit grounded evidence and producing fabricated responses when the retrieved context is insufficient. While prior research has addressed these issues independently, a unified framework that integrates evidence-based grounding and reliable abstention is currently lacking. In this paper, we propose GRACE, a reinforcement-learning framework that simultaneously mitigates both types of flaws. GRACE employs a data construction method that utilizes heterogeneous retrievers to generate diverse training samples without manual annotation. A multi-stage gated reward function is then employed to train the model to assess evidence sufficiency, extract key supporting evidence, and provide answers or explicitly abstain. Experimental results on two benchmarks demonstrate that GRACE achieves state-of-the-art overall accuracy and strikes a favorable balance between accurate response and rejection, while requiring only 10% of the annotation costs of prior methods. Our code is available at https://github.com/YiboZhao624/Grace..

Problem

• Retrieval-Augmented Generation
• evidence grounding
• abstention
• hallucination
• reinforcement learning

Innovation

• Reinforcement Learning
• Retrieval-Augmented Generation
• Evidence Grounding
• Abstention Mechanism
• Reward Shaping