PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generation

2026-06-15

AI Summary

This work addresses the “answer-path reward confusion” and “search-update ambiguity” in agent-based retrieval-augmented generation, which arise from relying solely on final-answer rewards. To resolve these issues, the authors propose PathRouter, a framework that jointly evaluates answer correctness and evidence-path overlap. It categorizes trajectories into four types and applies differentiated scaling to the GRPO advantage function accordingly. Additionally, PathRouter incorporates a frozen golden-evidence teacher model to impose token-level KL divergence regularization on non-answer tokens. This path-aware reinforcement learning mechanism effectively suppresses shortcut behaviors while preserving exploration capacity, yielding significant performance gains across six question-answering benchmarks: average F1 scores improve by 3.1 and 4.9 for 3B and 7B models, respectively, alongside enhanced alignment with evidence paths.

Abstract

Agentic GraphRAG trains language-model agents to iteratively retrieve and reason over graph-structured evidence, enabling more accurate and context-aware decision-making by efficiently navigating complex information networks. However, outcome-only reinforcement learning suffers from \textit{\textbf{answer-path reward aliasing}}, where correct answers may come from shortcuts rather than useful evidence paths. It also exhibits \textit{\textbf{search-update ambiguity}}, as scalar trajectory-level feedback does not indicate which retrieval actions to adjust. To mitigate these shortcomings, we present PathRouter, a path-aware training framework for agentic GraphRAG. PathRouter jointly evaluates each trajectory along answer correctness and evidence-path overlap, yielding four trajectory categories with differentiated GRPO advantage scaling that suppresses shortcut reinforcement while preserving evidence-seeking behavior. For evidence-poor trajectories, a frozen gold-evidence teacher provides token-level KL guidance on reasoning and search-query tokens, excluding answer tokens to avoid direct response imitation. Experiments on six QA benchmarks across three model sizes show that PathRouter consistently improves answer F1 and evidence-path overlap, achieving average F1 gains of 3.1 on 3B and 4.9 on 7B models compared to a strong baseline.

Problem

• reward aliasing
• search-update ambiguity
• graph retrieval-augmented generation
• evidence path
• reinforcement learning

Innovation

• PathRouter
• Agentic GraphRAG
• reward aliasing
• evidence-path overlap
• GRPO