ProGraph-R1: Progress-aware Reinforcement Learning for Graph Retrieval Augmented Generation

2026-01-25

AI Summary

This work addresses the limitations of existing reinforcement learning–based graph retrieval-augmented generation (GraphRAG) methods, which often neglect graph structural information and rely solely on sparse final rewards, thereby struggling to effectively optimize intermediate steps in multi-hop reasoning. To overcome these challenges, the authors propose a progress-aware agent framework that introduces a novel hypergraph retrieval mechanism jointly capturing semantic relevance and graph connectivity. Furthermore, they design a dense reward signal based on reasoning progress and employ a progress-aware advantage function for policy optimization. By deeply integrating reinforcement learning, large language models, and knowledge graphs, the proposed approach significantly improves both reasoning accuracy and generation quality on multi-hop question answering benchmarks, outperforming current GraphRAG methods.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Graph Retrieval-Augmented Generation (GraphRAG) has been successfully applied in various knowledge-intensive question answering tasks by organizing external knowledge into structured graphs of entities and relations. It enables large language models (LLMs) to perform complex reasoning beyond text-chunk retrieval. Recent works have employed reinforcement learning (RL) to train agentic GraphRAG frameworks that perform iterative interactions between LLMs and knowledge graphs. However, existing RL-based frameworks such as Graph-R1 suffer from two key limitations: (1) they primarily depend on semantic similarity for retrieval, often overlooking the underlying graph structure, and (2) they rely on sparse, outcome-level rewards, failing to capture the quality of intermediate retrieval steps and their dependencies. To address these limitations, we propose ProGraph-R1, a progress-aware agentic framework for graph-based retrieval and multi-step reasoning. ProGraph-R1 introduces a structure-aware hypergraph retrieval mechanism that jointly considers semantic relevance and graph connectivity, encouraging coherent traversal along multi-hop reasoning paths. We also design a progress-based step-wise policy optimization, which provides dense learning signals by modulating advantages according to intermediate reasoning progress within a graph, rather than relying solely on final outcomes. Experiments on multi-hop question answering benchmarks demonstrate that ProGraph-R1 consistently improves reasoning accuracy and generation quality over existing GraphRAG methods.

Problem

• Graph Retrieval-Augmented Generation
• reinforcement learning
• graph structure
• multi-hop reasoning
• sparse rewards

Innovation

• structure-aware retrieval
• progress-based reinforcement learning
• hypergraph traversal
• step-wise policy optimization
• multi-hop reasoning